Estimating shelf capacity from image data to improve store execution

ABSTRACT

Systems and methods for automatically determining options for store execution based on shelf capacity are provided. In one implementation, at least one processor is configured to receive a set of images depicting a first plurality of products from a particular product type displayed on a shelving unit in a retail store; analyze the set of images to identify a portion of the shelving unit associated with the particular product type; determine a product capacity for the portion of the shelving unit; access stored data about a second plurality of products from the particular product type, wherein each of the second plurality of products is located separately from the shelving unit; use the product capacity for the portion of the shelving unit and the accessed data to generate a suggestion for improving store execution; and provide the at least one suggestion for improving store execution.

CROSS REFERENCES TO RELATED APPLICATIONS

This application claims the benefit of priority of U.S. ProvisionalApplication No. 62/814,339, filed Mar. 6, 2019; U.S. ProvisionalApplication No. 62/829,160, filed Apr. 4, 2019; U.S, ProvisionalApplication No. 62/872,751, filed Jul. 11, 2019; and U.S. ProvisionalApplication No. 62/879,565, filed Jul. 29, 2019. The foregoingapplications are incorporated herein by reference in their entirety.

BACKGROUND I. Technical Field

The present disclosure relates generally to systems, methods, anddevices for identifying products in retail stores, and more specificallyto systems, methods, and devices for capturing, collecting, andautomatically analyzing images of products displayed in retail storesfor purposes of providing one or more functions associated with theidentified products.

II. Background Information

Shopping in stores is a prevalent part of modern daily life. Storeowners (also known as “retailers”) stock a wide variety of products onstore shelves and add associated labels and promotions to the storeshelves. Typically, retailers have a set of processes and instructionsfor organizing products on the store shelves. The source of some ofthese instructions may include contractual obligations and otherpreferences related to the retailer methodology for placement ofproducts on the store shelves. Nowadays, many retailers and supplierssend people to stores to personally monitor compliance with the desiredproduct placement. Such a monitoring technique, however, may beinefficient and may result in nonuniform compliance among retailersrelative to various product-related guidelines. This technique may alsoresult in significant gaps in compliance, as it does not allow forcontinuous monitoring of dynamically changing product displays. Toincrease productivity, among other potential benefits, there is atechnological need to provide a dynamic solution that will automaticallymonitor retail spaces. Such a solution, for example and among otherfeatures, may automatically determine whether a disparity exists betweena desired product placement and an actual product placement.

The disclosed devices and methods are directed to providing new ways formonitoring retail establishments using image processing and supportingsensors.

SUMMARY

Embodiments consistent with the present disclosure provide systems,methods, and devices For capturing, collecting, and analyzing images ofproducts displayed in retail stores. For example, consistent with thedisclosed embodiments, an example system may receive an image depictinga store shelf having products displayed thereon, identify the productson the store shelf, and trigger an alert when disparity exists betweenthe desired product placement and the actual product placement.

Another aspect of the present disclosure is directed to a computerprogram product for identifying products and monitoring planogramcompliance using analysis of image data embodied in a non-transitorycomputer-readable medium and executable by at least one processor, thecomputer program product including instructions for causing the at leastone processor to execute the method described above.

In an embodiment, a computer-implemented system for processing imagescaptured in a retail store to determine tasks for non-employeeindividuals associated with the retail store may include at least oneprocessor configured to receive one or more images captured in a retailstore and depicting a plurality of products displayed on at least onestore shelf. The at least processor may also be configured to analyzethe one or more images to determine at least one task associated with astore shelf and determine an incentive for completing the at least onetask based on a property of the store shelf. The incentive may beintended for a non-employee individual associated with the retail store.The at least one processor may further be configured to provide an offerto at least one individual non-employee individual associated with theretail store. The offer may include receiving the incentive in responseto completing the at least one task.

In an embodiment, a computer program product for processing imagescaptured in a retail store to determine tasks for non-employeeindividuals associated with the retail store is provided. The computerprogram product may be embodied in a non-transitory computer-readablemedium and executable by at least one processor. The computer programproduct may include instructions for causing the at least one processorto execute a method including receiving one or more images captured in aretail store and depicting a plurality of products displayed on at leastone store shelf The method may also include analyzing the one or moreimages to determine at least one task associated with a store shelf anddetermining an incentive for completing the at least one task based on aproperty of the store shelf The incentive may be intended for anon-employee individual associated with the retail store. The method mayfurther include providing an offer to at least one non-employeeindividual associated with the retail store, and the offer may includereceiving the incentive upon completing the at least one task.

In an embodiment, a method for processing images captured in a retailstore to determine tasks for non-employee individuals associated withthe retail store may include receiving one or more images captured in aretail store and depicting a plurality of products displayed on at leastone store shelf. The method may also include analyzing the one or moreimages to determine at least one task associated with a store shelf anddetermining an incentive for completing the at least one task based on aproperty of the store shelf. The incentive may be intended for anon-employee individual associated with the retail store. The method mayfurther include providing an offer to at least one non-employeeindividual associated with the retail store, and the offer may includereceiving the incentive upon completing the at least one task.

In an embodiment, a computer-implemented system for identifyingperishable products in a retail store based on analysis of image dataand for automatically generating offers relating to the identifiedproducts is provided. The system may include at least one processorconfigured to receive a set of images depicting a plurality ofperishable products of at least one product type displayed on at leastone shelving unit in a retail store. The at least one processor may alsobe configured to analyze the set of images to determine informationabout a displayed inventory of the plurality of perishable products andaccess a database storing information about the at least one producttype. The at least one processor may further be configured to use theinformation about the displayed inventory and the stored informationabout the at least one product type to determine at least one offerassociated with the at least one product type and provide the at leastone offer to at least one costumer of the retail store.

In one embodiment, a computer program product for identifying perishableproducts in a retail store based on analysis of image data and forautomatically generating offers relating to the identified productsembodied in a non-transitory computer-readable medium and executable byat least one processor is provided. The computer program productincludes instructions for causing the at least one processor to executea method including receiving a set of images depicting a plurality ofperishable products of at least one product type displayed on at leastone shelving unit in a retail store. The method may also includeanalyzing the set of images to determine information about a displayedinventory of the plurality of perishable products and accessing adatabase storing information about the at least one product type. Themethod may further include using the information about the displayedinventory and the stored information about the at least one product typeto determine at least one offer associated with the at least one producttype, and providing the at least one offer to at least one costumer ofthe retail store.

In an embodiment, a method for identifying perishable products in aretail store based on analysis of image data and for automaticallygenerating offers relating to the identified products is provided. Themethod may include receiving a set of images depicting a plurality ofperishable products of at least one product type displayed on at leastone shelving unit in a retail store. The method may also includeanalyzing the set of images to determine information about a displayedinventory of the plurality of perishable products and accessing adatabase storing information about the at least one product type. Themethod may further include using the information about the displayedinventory and the stored information about the at least one product typeto determine at least one offer associated with the at least one producttype, and providing the at least one offer to at least one costumer ofthe retail store.

In an embodiment, a computer-implemented system for processing imagescaptured in a retail store and automatically identifying changes inplanograms is provided. The system may include at least one processorconfigured to access a plurality of planograms stored in a database.Each planogram may describe a desired placement of products on shelvesof a retail store during a time period. The plurality of planograms mayinclude at least a first planogram and a second planogram. The at leastone processor may also be configured to receive a first set of imagescaptured at a first time and depicting a first plurality of productsdisplayed on at least one of the shelves of the retail store. The atleast one processor may further be configured to analyze the first setof images to determine an actual placement of the first plurality ofproducts displayed on the shelves of the retail store at the first time.The at least one processor may also be configured to identify adeviation of an actual placement of at least some of the first pluralityof products from the desired placement of products associated with thefirst planogram. The at least one processor may further be configured toissue a first user-notification associated with the deviation of theactual placement of the at least some of the first plurality of productsfrom the desired placement of products associated with the firstplanogram. The at least one processor may also be configured to afterissuing the first user-notification, receive a second set of imagescaptured at a second time and depicting a second plurality of productsdisplayed on the at least one of the shelves of the retail store. The atleast one processor may further be configured to analyze the second setof images to determine an actual placement of the second plurality ofproducts displayed on the shelves of the retail store at the secondtime. The at least one processor may also be configured to identify adeviation of the actual placement of at least some of the secondplurality of products from the desired placement of products associatedwith the first planogram. The at least one processor may further beconfigured to use the second planogram to determine whether anarrangement associated with the second plurality of products conforms tothe second planogram rather than to the first planogram. The at leastone processor may also be configured to, when the arrangement associatedwith the second plurality of products conforms to the second planogram,withhold issuance of a second user-notification indicating a deviationrelative to the first planogram.

Juan embodiment, a computer program product for processing imagescaptured in a retail store and automatically identifying changes inplanograms, which may be embodied in a non-transitory computer-readablemedium and executable by at least one processor, is provided. Thecomputer program product includes instructions for causing the at leastone processor to execute a method including accessing a plurality ofplanograms stored in a database. Each planogram may describe a desiredplacement of products on shelves of a retail store during a time period.The plurality of planograms may further include at least a firstplanogram and a second planogram. The method may also include receivinga first set of images captured at a first time and depicting a firstplurality of products displayed on at least one of the shelves of theretail store. The method may further include analyzing the first set ofimages to determine an actual placement of the first plurality ofproducts displayed on the shelves of the retail store at the first time.The method may also include identifying a deviation of an actualplacement of at least some of the first plurality of products from thedesired placement of products associated with the first planogram. Themethod may further include issuing a first user-notification associatedwith the deviation of the actual placement of the at least some of thefirst plurality of products from the desired placement of productsassociated with the first planogram. The method may also include afterissuing the first user-notification, receiving a second set of imagescaptured at a second time and depicting a second plurality of productsdisplayed on the at least one of the shelves of the retail store. Themethod may further include analyzing the second set of images todetermine an actual placement of the second plurality of productsdisplayed on the shelves of the retail store at the second time. Themethod may also include identifying a deviation of the actual placementof at least some of the second plurality of products from the desiredplacement of products associated with the first planogram. The methodmay further include using the second planogram to determine whether anarrangement associated with the second plurality of products conforms tothe second planogram rather than to the first planogram. The method mayalso include when the arrangement associated with the second pluralityof products conforms to the second planogram, withholding issuance of asecond user-notification indicating a deviation relative to the firstplanogram.

In an embodiment, a method for processing images captured in a retailstore and automatically identifying changes in planograms may includeaccessing a plurality of planograms stored in a database. Each planogrammay describe a desired placement of products on shelves of a retailstore during a time period. The plurality of planograms may furtherinclude at least a first planogram and a second planogram. The methodmay also include receiving a first set of images captured at a firsttime and depicting a first plurality of products displayed on at leastone of the shelves of the retail store. The method may further includeanalyzing the first set of images to determine an actual placement ofthe first plurality of products displayed on the shelves of the retailstore at the first time. The method may also include identifying adeviation of an actual placement of at least some of the first pluralityof products from the desired placement of products associated with thefirst planogram. The method may further include issuing a firstuser-notification associated with the deviation of the actual placementof the at least some of the first plurality of products from the desiredplacement of products associated with the first planogram. The methodmay also include after issuing the first user-notification, receiving asecond set of images captured at a second time and depicting a secondplurality of products displayed on the at least one of the shelves ofthe retail store. The method may further include analyzing the secondset of images to determine an actual placement of the second pluralityof products displayed on the shelves of the retail store at the secondtime. The method may also include identifying a deviation of the actualplacement of at least some of the second plurality of products from thedesired placement of products associated with the first planogram. Themethod may further include using the second planogram to determinewhether an arrangement associated with the second plurality of productsconforms to the second planogram rather than to the first planogram. Themethod may also include when the arrangement associated with the secondplurality of products conforms to the second planogram, withholdingissuance of a second user-notification indicating a deviation relativeto the first planogram.

In an embodiment, an apparatus may secure a camera to a retail shelvingunit. The apparatus may include: a pair of opposing securing surfacesfor securing the apparatus to a support element of a retail shelvingunit, wherein the pair of opposing securing surfaces forms a gap betweenthe pair of opposing securing surfaces, wherein the gap is configured toaccept the support element of the retail shelving unit; and an armextending from one of the pair of opposing securing surfaces and havinga distal end, the distal end including a camera support area includingat least one opening.

In an embodiment, a system may monitor planogram compliance in a retailstore. The system may include: a plurality of cameras configured tocapture images depicting a plurality of products displayed in the retailstore, wherein each camera is mountable to a retail shelving unit usingan apparatus including: a pair of opposing securing surfaces forsecuring the apparatus to a support element of the retail shelving unit,wherein the pair of opposing securing surfaces forms a gap between thepair of opposing securing surfaces, and wherein the gap is configured toaccept the support element of the retail shelving unit; and an armextending from one of the pair of opposing securing surfaces and havinga distal end, the distal end including a camera support area including aleast one opening; a data interface configured to receive image datafrom the plurality of cameras; and at least one processor for monitoringplanogram compliance using the received image data.

In an embodiment, a system may process images captured in a retail storeand automatically determining relationships between products and shelflabels. The system may include at least one processor configured to:receive images of at least one store shelf captured over a period oftime by one or more image sensors, wherein the images depict a pluralityof shelf labels and groups of products associated with different producttypes placed on the at least one store shelf analyze the images toidentify temporal changes in an arrangement of the groups of productsover the period of time, wherein at a first point in time the groups ofproducts are arranged on the least one store shelf in a first manner andat a second point in time the groups of products are arranged on theleast one store shelf in a second manner; analyze the images to identifytemporal changes in display of the plurality of shelf labels over theperiod of time, wherein at the first point in time the plurality ofshelf labels are displayed on the least one store shelf in a firstmanner and at the second point in time the plurality of shelf labels aredisplayed on the least one store shelf in a second manner; determine arelationship between a specific group of products and a specific shelflabel based on the identified temporal changes in the arrangement of thegroups of products and the identified temporal changes in the display ofthe plurality of shelf labels; and initiate an action based on thedetermined relationship.

In an embodiment, a computer program product may process images capturedin a retail store and automatically determine relationships betweenproducts and shelf labels. The computer program product may be embodiedin a non-transitory computer-readable medium and may includeinstructions for causing at least one processor to execute a method. Themethod may include: receiving images of at least one store shelfcaptured over a period of time by one or more image sensors, wherein theimages depict a plurality of shelf labels and groups of productsassociated with different product types placed on the at least one storeshelf; analyzing the images to identify temporal changes in arrangementof the groups of products over the period of time, wherein at a firstpoint in time the groups of products are arranged on the least one storeshelf in a first manner and at a second point in time the groups ofproducts are arranged on the least one store shelf in a second manner;analyzing the images to identify temporal changes in display of theplurality of shelf labels over the period of time, wherein at the firstpoint in time the plurality of shelf labels are displayed on the leastone store shelf in a first manner and at the second point in time theplurality of shelf labels are displayed on the least one store shelf ina second manner; determining a relationship between a specific group ofproducts placed on the at least one store shelf and a specific shelflabel based on the identified temporal changes in the arrangement of thegroups of products and the identified temporal changes in the display ofthe plurality of shelf labels; and initiating an action based on thedetermined relationship.

In an embodiment, a method may process images captured in a retail storeand automatically determine relationships between products and shelflabels. The method may include: receiving images of at least one storeshelf captured over a period of time by one or more image sensors,wherein the images depict a plurality of shelf labels and groups ofproducts associated with different product types placed on the at leastone store shelf; analyzing the images to identify temporal changes inarrangement of the groups of products over the period of time, whereinat a first point in time the groups of products are arranged on theleast one store shelf in a first manner and at a second point in timethe groups of products are arranged on the least one store shelf in asecond manner; analyzing the images to identify temporal changes indisplay of the plurality of shelf labels over the period of time,wherein at the first point in time the plurality of shelf labels aredisplayed on the least one store shelf in a first manner and at thesecond point in time the plurality of shelf labels are displayed on theleast one store shelf in a second manner; determining a relationshipbetween a specific group of products placed on the at least one storeshelf and a specific shelf label based on the identified temporalchanges in the arrangement of the groups of products and the identifiedtemporal changes in the display of the plurality of shelf labels; andinitiating an action based on the determined relationship.

In an embodiment, a system may process images captured in a retail storeand automatically identify situations to withhold planogram incompliancenotifications. The system may include at least one processor configuredto: receive image data from a plurality of image sensors mounted in theretail store, wherein the image data depicts a plurality of productsdisplayed on at least one store shelf; analyze the image data toidentify at least one product type associated with the plurality ofproducts displayed on at least one store shelf and to determine aplacement of products of the at least one product type on the at leastone store shelf; access at least one planogram describing a desiredplacement of products of the at least one product type on shelves of aretail store; identify, based on the at least one planogram, adiscrepancy between the determined placement of products associated withthe identified at least one product type and the desired placement ofproducts associated with the identified at least one product type; anddetermine whether a notice-override condition exists relative to theidentified at least one product type due to misidentification of theproduct. If a notice-override condition exists relative to theidentified at least one product type, the at least one processor may beconfigured to withhold issuance of a user-notification associated withthe identified discrepancy between the determined placement of productsand the desired placement of products. If a notice-override condition isnot determined to exist relative to the identified at least one producttype, the at least one processor may be configured to issue auser-notification associated with the identified discrepancy between thedetermined placement of products and the desired placement of products.

In an embodiment, a computer program product may process images capturedin a retail store and automatically withhold false planogramincompliance notifications. The computer program product may be embodiedin a non transitory computer-readable medium and including instructionsfor causing at least one processor to execute a method. The method mayinclude: receiving image data from a plurality of image sensors mountedin the retail store, wherein the image data depicts a plurality ofproducts displayed on at least one store shelf; analyzing the image datato identify at least one product type associated with the plurality ofproducts displayed on at least one store shelf and to determine aplacement of products of the at least one product type on the at leastone store shelf; accessing at least one planogram describing a desiredplacement of products of the at least one product type on shelves of aretail store; identifying, based on the at least one planogram, adiscrepancy between the determined placement of products associated withthe identified at least one product type and the desired placement ofproducts associated with the identified at least one product type; anddetermining whether a notice-override condition exists relative to theidentified at least one product type due to misidentification of theproduct. If a notice-override condition exists relative to theidentified at least one product type, the method may include withholdingissuance of a user-notification associated with the identifieddiscrepancy between the determined placement of products and the desiredplacement of products. If a notice-override condition is not determinedto exist relative to the identified at least one product type, themethod may include issuing a user-notification associated with theidentified discrepancy between the determined placement of products andthe desired placement of products.

In an embodiment, a method may process images captured in a retail storeand automatically identify situations to withhold planogram incompliancenotifications. The method may include: receiving image data from aplurality of image sensors mounted in the retail store, wherein theimage data depicts a plurality of products displayed on at least onestore shelf; analyzing the image data to identify at least one producttype associated with the plurality of products displayed on at least onestore shelf and to determine a placement of products of the at least oneproduct type on the at least one store shelf; accessing at least oneplanogram describing a desired placement of products of the at least oneproduct type on shelves of a retail store; identifying, based on the atleast one planogram, a discrepancy between the determined placement ofproducts associated with the identified at least one product type andthe desired placement of products associated with the identified atleast one product type; and determining whether a notice-overridecondition exists relative to the identified at least one product typedue to misidentification of the product. If a notice-override conditionexists relative to the identified at least one product type, the methodmay include withholding issuance or a user-notification associated withthe identified discrepancy between the determined placement of productsand the desired placement of products. If a notice-override condition isnot determined to exist relative to the identified at least one producttype, the method may include issuing a user-notification associated withthe identified discrepancy between the determined placement of productsand the desired placement of products.

In an embodiment, a mobile power source may comprise a housingconfigured to retain at least one battery. The housing may include afirst guiding channel located on a first side of the housing and asecond guiding channel located on a second side of the housing oppositeto the first side of the housing. The first and second guiding channelsmay be configured to engage with a track associated with a retailshelving unit in order to secure the housing to the retail shelvingunit. The mobile power source may further comprise a first electricalconnector associated with the housing. The first connector may beconfigured to electrically connect the mobile power source to a controlunit for at least one image capture device. The mobile power source mayfurther comprise a second electrical connector associated with thehousing. The second electrical connector may be configured toelectrically connect the mobile power source to at least one of anadditional mobile power source or a power grid. The mobile power sourcemay also comprise circuitry configured to convey power from theadditional mobile power source to the at least one image capture devicewhen the mobile power source is connected to the additional mobile powersource, and to convey power from the power grid to the at least oneimage capture device when the mobile power source is connected to thepower grid.

In an embodiment, a control unit for a plurality of image capturedevices may comprise a housing including a first guiding channel locatedon a first side of the housing and a second guiding channel located on asecond side of the housing opposite to the first side of the housing.The first and second guiding channels may be configured to slidablyengage with a track associated with a retail shelving unit. The controlunit may further comprise a plurality of connectors located on a firstend of the housing. Each connector may be configured to enable aconnection for transferring power to and for receiving data from aseparate image capture device fixedly mounted to the retail shelvingunit. The control unit may further comprise a power port on a second endof the housing opposite to the first end of the housing. The power portmay be configured to enable an electrical connection for receiving powerfrom at least one mobile power source.

In an embodiment, an apparatus for slidably securing a selectable numberof mobile power sources to a retail shelving unit for supplying power toat least one image capture device may comprise a base configured to bemounted on a first surface of the retail shelving unit opposite to asecond surface of the retail shelving unit designated for placement ofretail products. The apparatus may comprise a first rail arm extendingfrom the base. The first rail arm may include a first rail protrudingfrom the first rail arm and extending in a longitudinal direction alongthe first rail arm. The apparatus may also comprise a second rail armextending from the base. The second rail arm may include a second railprotruding from the second mil arm in a direction toward the first railarm and extending in a longitudinal direction along the second rail arm.The first and second rails may be configured to engage withcorresponding grooves of one or more mobile power sources to slidablysecure the one or more power sources to the retail shelving unit.

In an embodiment, a system for processing images captured in multipleretail stores and automatically coordinating actions across two or moreretail stores may comprise at least one processor. The at least oneprocessor may be configured to receive first image data captured from anenvironment of a first retail store depicting products displayed onshelves of the first retail store and analyze the first image data toidentify a first planogram incompliance event associated with at leastone shelf of the first retail store. The at least one processor may thengenerate an instruction for an employee of the first retail store toperform a remedial action in response to the identified first planogramincompliance event and acquire and store feedback from the employee ofthe first retail store regarding the first planogram incompliance event.The at least one processor may further receive second image datacaptured from an environment of a second retail store depicting productsdisplayed on shelves of the second retail store and analyze the secondimage data to identify a second planogram incompliance event associatedwith at least one shelf of the second retail store. The at least oneprocessor may determine that the at least one shelf of the second retailstore is related to the at least one shelf of the first retail store,access the feedback from the employee of the first retail storeregarding the first planogram incompliance event associated with the atleast one shelf of the first retail store, and use the feedback from theemployee in determining what action to take in response to the secondplanogram incompliance event associated with at least one shelf of thesecond retail store. The at least one processor may then initiate anaction based on the determination.

In an embodiment, a computer program product for processing imagescaptured in multiple retail stores and automatically coordinatingactions across two or more retail stores may be embodied in anon-transitory computer-readable medium and may include instructions forcausing at least one processor to execute a method. The method maycomprise receiving first image data captured from an environment of afirst retail store depicting products displayed on shelves of the firstretail store and analyzing the first image data to identify a firstplanogram incompliance event associated with at least one shelf of thefirst retail store. The method may further comprise generating aninstruction for an employee of the first retail store to perform aremedial action in response to the identified first planogramincompliance event and acquiring and storing feedback from the employeeof the first retail store regarding the first planogram incomplianceevent. The method may comprise receiving second image data captured froman environment of a second retail store depicting products displayed onshelves of the second retail store and determining that the at least oneshelf of the second retail store is related to at least one shelf of thefirst retail store. The method may further comprise determining that theat least one shelf of the second retail store is related to at least oneshelf of the first retail store, accessing the feedback from theemployee of the first retail store regarding the first planogramincompliance event associated with the at least one shelf of the firstretail store, and using the feedback from the employee of the firstretail store in determining what action to take in response to thesecond planogram incompliance event associated with at least one shelfof the second retail store. The method may then comprise initiating anaction based on the determination.

In an embodiment, a method for processing images captured in multipleretail stores and automatically coordinating actions across two or moreretail stores may comprise receiving first image data captured from anenvironment of a first retail store depicting products displayed onshelves of the first retail store and analyzing the first image data toidentify a first planogram incompliance event associated with at leastone shelf of the first retail store. The method may further comprisegenerating an instruction for an employee of the first retail store toperform a remedial action in response to the identified first planogramincompliance event and acquiring and storing feedback from the employeeof the first retail store regarding the first planogram incomplianceevent. The method may comprise receiving second image data captured froman environment of a second retail store depicting products displayed onshelves of the second retail store and determining that the at least oneshelf of the second retail store is related to at least one shelf of thefirst retail store. The method may further comprise determining that theat least one shelf of the second retail store is related to at least oneshelf of the first retail store, accessing the feedback from theemployee of the first retail store regarding the first planogramincompliance event associated with the at least one shelf of the firstretail store, and using the feedback from the employee of the firstretail store in determining what action to take in response to thesecond planogram incompliance event associated with at least one shelfof the second retail store. The method may then comprise initiating anaction based on the determination.

In an embodiment, a system processing images captured in a retail storeand automatically determining options for store execution based on shelfcapacity may comprise at least one processor. The at least one processormay be configured to receive a set of images captured during a period oftime, wherein the set of images depict a first plurality of productsfrom a particular product type displayed on a shelving unit in a retailstore, and analyze the set of images to identify a portion of theshelving unit associated with the particular product type. The at leastone processor may further determine a product capacity for the portionof the shelving unit dedicated to the particular product type, whereinthe determined product capacity is greater than a number of the firstplurality of products depicted in the set of images and access storeddata about a second plurality of products from the particular producttype, wherein each of the second plurality of products is locatedseparately from the shelving unit when the period of time ends. The atleast one processor may then use the product capacity for the portion ofthe shelving unit dedicated to the particular product type and theaccessed data to generate at least one suggestion for improving storeexecution and provide the at least one suggestion for improving storeexecution.

In an embodiment, a computer program product for processing imagescaptured in a retail store and automatically determining options forstore execution based on shelf capacity may be embodied in anon-transitory computer-readable medium and may include instructions forcausing at least one processor to execute a method. The method maycomprise receiving a set of images captured during a period of time,wherein the set of images depicts a first plurality of products from aparticular product type displayed on a shelving unit in a retail store,and analyzing the set of images to identify a portion of the shelvingunit dedicated to the particular product type. The method may furthercomprise determining a product capacity for the portion of the shelvingunit dedicated to the particular product type, wherein the determinedproduct capacity is greater than a number of the first plurality ofproducts depicted in the set of images, and accessing stored data abouta second plurality of products from the particular product type, whereineach of the second plurality of products is located separately from theshelving unit when the period of time ends. The method may then compriseusing the determined product capacity and the accessed data to generateat least one suggestion for improving store execution and providing theat least one suggestion for improving store execution.

In an embodiment, a method for processing images captured in a retailstore and automatically determining options for store execution based onshelf capacity may comprise receiving a set of images captured during aperiod of time, wherein the set of images depicts a first plurality ofproducts from a particular product type displayed on a shelving unit ina retail store, and analyzing the set of images to identify a portion ofthe shelving unit dedicated to the particular product type. The methodmay further comprise determining a product capacity for the portion ofthe shelving unit dedicated to the particular product type, wherein thedetermined product capacity is greater than a number of the firstplurality of products depicted in the set of images, and accessingstored data about a second plurality of products from the particularproduct type, wherein each of the second plurality of products islocated separately from the shelving unit when the period of time ends.The method may then comprise using the determined product capacity andthe accessed data to generate at least one suggestion for improvingstore execution and providing the at least one suggestion for improvingstore execution.

In an embodiment, a system may process images captured in a retail storeand automatically generate low-stock alerts. The system may include atleast one processor configured to receive image data associated withimages captured during a first period of time by one or more imagecapturing devices mounted in the retail store. The image data may depictinventory changes in at least a portion of a retail shelving unitdedicated to products from a specific product type. The at least oneprocessor may analyze the image data to identify a restocking eventassociated with the at least a portion of the retail shelving unit. Theat least one processor may also analyze the image data to identify atleast one facing event associated with the at least a portion of theretail shelving unit, wherein the at least one facing event occurs afterthe restocking event. The at least one processor may further analyze theimage data to identify a low-stock event associated with the at least aportion of the retail shelving unit, wherein the low-stock event occursafter the at least one facing event. Thereafter, the at least oneprocessor may further determine a demand pattern for products from thespecific product type based on the identified restocking event, the atleast one facing event, and the low-stock event. The at least oneprocessor may further receive additional image data associated withimages captured during a second period of time, subsequent to thelow-stock event, by the plurality of image capturing devices. Theadditional image data may depict additional inventory changes in the atleast a portion of the retail shelving unit. Subsequently, the at leastone processor may predict when a next low-stock event will occur basedon the additional image data and the determined demand pattern, andgenerate an alert associated with the predicted next low-stock event.

In an embodiment, a computer program product for processing imagescaptured in a retail store may automatically generate low-stock alerts.The computer program product may be embodied in a non-transitorycomputer-readable medium and include instructions for causing at leastone processor to execute a method comprising: receiving image dataassociated with images captured during a first period of time by one ormore image capturing devices mounted in the retail store, wherein thecaptured data depicts inventory changes in at least a portion of aretail shelving unit dedicated to products from a specific product type;analyzing the image data to identify a restocking event associated withthe at least a portion of the retail shelving unit; analyzing the imagedata to identify at least one facing event associated with the at leasta portion of the retail shelving unit, wherein the at least one facingevent occurs after the restocking event; analyzing the image data toidentify a low-stock event associated with the at least a portion of theretail helving unit, wherein the low-stock event occurs after the atleast one facing event; determining a demand pattern for products fromthe specific product type based on the identified restocking event, theat least one facing event, and the low-stock event; receiving additionalimage data associated with images captured during a second period oftime subsequent to the low-stock event by the plurality of imagecapturing devices, wherein the additional image data depicts additionalinventory changes in the at least a portion of the retail shelving unit;based on the additional image data and the determined demand pattern,predicting when a next low-stock event will occur; and generating analert associated with the predicted next low-stock event.

In an embodiment, a method may process images captured in a retail storeand automatically generate low-stock alerts. The method may comprise:receiving image data associated with images captured during a firstperiod of time by one or more image capturing devices mounted in theretail store, wherein the captured data depicts inventory changes in atleast a portion of a retail shelving unit dedicated to products from aspecific product type; analyzing the image data to identify a restockingevent associated with the at least a portion of the retail shelvingunit; analyzing the image data to identify at least one facing eventassociated with the at least a portion of the retail shelving unit,wherein the at least one facing event occurs after the restocking event;analyzing the image data to identify a low-stock event associated withthe at least a portion of the retail shelving unit, wherein thelow-stock event occurs after the at least one facing event; determininga demand pattern for products from the specific product type based onthe identified restocking event, the at least one facing event, and thelow-stock event; receiving additional image data associated with imagescaptured during a second period of time subsequent to the low-stockevent by the plurality of image capturing devices, wherein theadditional image data depicts additional inventory changes in the atleast a portion of the retail shelving unit; based on the additionalimage data and the determined demand pattern, predicting when a nextlow-stock event will occur; and generating an alert associated with thepredicted next low-stock event.

In an embodiment, a system for processing images captured in a retailstore and automatically identifying products displayed on shelving unitsmay include at least one processor configured to receive a first imagedepicting a shelving unit with a plurality of products of differingproduct types displayed thereon and a first plurality of labels coupledto the shelving unit. Each label depiction in the first image may haveat most a first image resolution. The at least one processor may also beconfigured to receive a set of second images depicting a secondplurality of labels. Each label depiction in the set of second imagesmay have at least a second image resolution greater than the first imageresolution. The at least one processor may also be configured tocorrelate a first label depiction of a specific label included in thefirst image to a second label depiction of the same specific labelincluded in a second image. The second image may be included in the setof second images. The at least one processor may also be configured touse information derived from the second label depiction to determine atype of products displayed in the first image in proximity to thespecific label, and initiate an action based on the determined type ofproducts displayed in proximity to the specific label.

In an embodiment, a computer program product for processing imagescaptured in a retail store and automatically identifying productsdisplayed on shelving units is provided. The computer program productmay be embodied in a non-transitory computer-readable medium andincluding instructions for causing at least one processor to execute amethod include receiving a first image depicting a shelving unit with aplurality of products of differing product types displayed thereon and afirst plurality of labels coupled to the shelving unit Each labeldepiction in the first image has at most a first image resolution. Themethod may also include receiving a set of second images depicting asecond plurality of labels. Each label depiction in the set of secondimages has at least a second image resolution greater than the firstimage resolution. The method may also include correlating a first labeldepiction of a specific label included in the first image to a secondlabel depiction of the same specific label included in a second image.The second image may be included in the set of second images. The methodmay also include using information derived from the second labeldepiction to determine a type of products displayed in the first imagein proximity to the specific label, and initiating an action based onthe determined type of products displayed in proximity to the specificlabel.

In an embodiment, a method for processing images captured in a retailstore and automatically identifying products displayed on shelving unitsmay include receiving a first image depicting a shelving unit with aplurality of products of differing product types displayed thereon and afirst plurality of labels coupled to the shelving unit. Each labeldepiction the first image may have at most a first image resolution. Themethod may also include receiving a set of second images depicting asecond plurality of labels. Each label depiction in the set of secondimages may have at least a second image resolution greater than thefirst image resolution. The method may also include correlating a firstlabel depiction of a specific label included in the first image to asecond label depiction of the same specific label included in a secondimage. The second image may be included in the set of second images. Themethod may further include using information derived from the secondlabel depiction to determine a type of products displayed in the firstimage in proximity to the specific label, and initiating an action basedon the determined type of products displayed in proximity to thespecific label.

Consistent with other disclosed embodiments, non-transitorycomputer-readable storage media may store program instructions, whichare executed by at least one processing device and perform any of themethods described herein.

The foregoing general description and the following detailed descriptionare exemplary and explanatory only and are not restrictive of theclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this disclosure, illustrate various disclosed embodiments. Inthe drawings:

FIG. 1 is an illustration of an exemplary system for analyzinginformation collected from a retail store.

FIG. 2 is a block diagram that illustrates some of the components of animage processing system, consistent with the present disclosure.

FIG. 3 is a block diagram that illustrates an exemplary embodiment of acapturing device, consistent with the present disclosure.

FIG. 4A is a schematic illustration of an example configuration forcapturing image data in a retail store, consistent with the presentdisclosure.

FIG. 4B is a schematic illustration or another example configuration forcapturing image data in a retail store, consistent with the presentdisclosure.

FIG. 4C is a schematic illustration of another example configuration forcapturing image data in a retail store, consistent with the presentdisclosure.

FIG. 5A is an illustration of an example system for acquiring images ofproducts in a retail store, consistent with the present disclosure.

FIG. 5B is an illustration of a shelf-mounted camera unit included in afirst housing of the example system of FIG. 5A, consistent with thepresent disclosure.

FIG. 5C is an exploded view illustration of a processing unit includedin a second housing of the example system of FIG. 5A, consistent withthe present disclosure.

FIG. 6A is a top view representation of an aisle in a retail store withmultiple image acquisition systems deployed thereon for acquiring imagesof products, consistent with the present disclosure.

FIG. 6B is a perspective view representation of part of a retailshelving unit with multiple image acquisition systems deployed thereonfor acquiring images of products, consistent with the presentdisclosure.

FIG. 6C provides a diagrammatic representation of how the exemplarydisclosed image acquisition systems may be positioned relative to retailshelving to acquire product images, consistent with the presentdisclosure.

FIG. 7A provides a flowchart of an exemplary method for acquiring imagesof products in retail store, consistent with the present disclosure.

FIG. 7B provides a flowchart of a method for acquiring images ofproducts in retail store, consistent with the present disclosure.

FIG. 8A is a schematic illustration of an example configuration fordetecting products and empty spaces on a store shell, consistent withthe present disclosure.

FIG. 8B is a schematic illustration of another example configuration fordetecting products and empty spaces on a store shelf, consistent withthe present disclosure.

FIG. 9 is a schematic illustration of example configurations fordetection elements on store shelves, consistent with the presentdisclosure.

FIG. 10A illustrates an exemplary method for monitoring planogramcompliance on a store shelf, consistent with the present disclosure.

FIG. 10B is illustrates an exemplary method for triggering imageacquisition based on product events on a store shelf, consistent withthe present disclosure.

FIG. 11A is a schematic illustration of an example output for a marketresearch entity associated with the retail store, consistent with thepresent disclosure.

FIG. 11B is a schematic illustration of an example output for a supplierof the retail store, consistent with the present disclosure.

FIG. 11C is a schematic illustration of an example output for a managerof the retail store, consistent with the present disclosure.

FIG. 11D is a schematic illustration of two examples outputs for anemployee of the retail store, consistent with the present disclosure.

FIG. 11E is a schematic illustration of an example output for an onlinecustomer of the retail store, consistent with the present disclosure.

FIG. 12 is a schematic illustration of an example system for determiningtasks for non-employee individuals associated with a retail store,consistent with the present disclosure.

FIGS. 13A and 13B are two example outputs for a non-employee of theretail store, consistent with the present disclosure.

FIG. 14 is a flowchart of a method for determining tasks fornon-employee individuals, consistent with the present disclosure.

FIG. 15 is a schematic illustration of an example system for generatingoffers relating to identified products, consistent with the presentdisclosure.

FIG. 16 is an example output for displaying an offer relating to aproduct, consistent with the present disclosure.

FIG. 17 is a flowchart of a method for generating an offer relating toan identified product, consistent with the present disclosure.

FIG. 18 is a schematic illustration of an example system for identifyingchanges in planograms, consistent with the present disclosure.

FIG. 19 is an example output for monitoring planogram compliance on astore shelf, consistent with the present disclosure.

FIG. 20A is a flowchart of an exemplary method for issuing anotification of planogram deviation, consistent with the presentdisclosure.

FIG. 20B is a flowchart of an exemplary method for issuing anotification of planogram deviation, consistent with the presentdisclosure.

FIG. 21A is a perspective view of an exemplary apparatus, consistentwith the present disclosure.

FIG. 21B is a side view of an exemplary apparatus, consistent with thepresent disclosure.

FIG. 21C is another exemplary apparatus, consistent with the presentdisclosure.

FIG. 21D is another exemplary apparatus, consistent with the presentdisclosure.

FIG. 22A is an illustration of an exemplary apparatus for securing acamera to a retail shelving unit, consistent with the presentdisclosure.

FIG. 22B is another illustration of an exemplary apparatus for securinga camera to a retail shelving unit, consistent with the presentdisclosure.

FIG. 23 is an illustration of a system for securing cameras to retailshelving units, consistent with the present disclosure.

FIG. 24 is a block diagram of a system for processing images captured ina retail store and automatically determining relationships betweenproducts and shelf labels, consistent with the present disclosure.

FIGS. 25A and 25B illustrate relationships between products and shelflabels, consistent with the present disclosure.

FIG. 26 illustrates a flowchart of an exemplary method for processingimages captured in a retail store and automatically determiningrelationships between products and shelf labels, consistent with thepresent disclosure.

FIG. 27 is a block diagram of a system for processing images captured ina retail store and automatically identifying situations to withholdplanogram incompliance notifications, consistent with the presentdisclosure.

FIG. 28A is an illustration of an image of a retail shelving unit,consistent with the present disclosure.

FIG. 28B is an illustration of a planogram, consistent with the presentdisclosure.

FIG. 29 is a flowchart of an exemplary method for processing imagescaptured in a retail store and automatically identifying situations towithhold planogram incompliance notifications, consistent with thepresent disclosure.

FIG. 30A is a schematic illustration of an example system for acquiringimages of products in a retail store, consistent with the presentdisclosure.

FIG. 30B is an illustration of an example control unit and mobile powersources, consistent with the present disclosure.

FIG. 30C illustrates an example retail shelving unit with a mobile powersources and a control unit slidably engaged along a track, consistentwith the present disclosure.

FIG. 31A is a side view illustrating an example apparatus for slidablysecuring a selectable number of mobile power sources to a retailshelving unit, consistent with the present disclosure.

FIGS. 31B and 31C show example electrical connectors for connecting amobile power source to a control unit and/or other mobile power sources,consistent with the present disclosure.

FIG. 32 provides a flowchart representing an exemplary method for usinga dynamic number of power sources for a camera mountable on a retailshelving unit, in accordance with the present disclosure.

FIG. 33 illustrates an example system for detecting planogram compliancein multiple retail stores, consistent with the present, disclosure.

FIG. 34 illustrates an example scenario for coordinating actions takenin response to identified incompliance events, consistent with thepresent disclosure.

FIG. 35 provides a flowchart representing an exemplary method forprocessing images captured in multiple retail stores and automaticallycoordinating actions across two or more retail stores, in accordancewith the present disclosure.

FIG. 36 illustrates an example environment for automatically determiningoptions for store execution based on shelf capacity, consistent with thepresent disclosure.

FIG. 37A illustrates example dimensions for a product that may bedetermined for estimating shelf capacity, consistent with the disclosedembodiments.

FIG. 37B illustrates various stored data that may be accessed forgenerating a suggestion for improving store execution, consistent withthe disclosed embodiments.

FIG. 38 provides a flowchart representing an exemplary method forprocessing images captured in a retail store and automaticallydetermining options for store execution based on shelf capacity,consistent with the disclosed embodiments.

FIGS. 39A-39F are illustrations of images captured in a retail storeconsistent with the present disclosure.

FIG. 40A includes two graphs showing example changes in facingdirections and quantities of products placed on a retail shelving unitconsistent with the present disclosure.

FIG. 40B is a block diagram illustrating an exemplary embodiment of amemory device containing software modules for executing methodsconsistent with the present disclosure.

FIG. 41 is a flowchart of an exemplary method for generating low-stockalerts consistent with the present disclosure.

FIG. 42 is a schematic illustration of an example system for processingimages captured in a retail store to automatically identify productsdisplayed on shelving units, consistent with the embodiments of thepresent disclosure.

FIG. 43A is a schematic illustration of an exemplary first image,consistent with the embodiments of the present disclosure.

FIG. 43B is a schematic illustration of an exemplary second image,consistent with the embodiments of the present disclosure.

FIG. 44 is a flowchart representing an exemplary method for processingimages captured in a retail store, consistent with embodiments of thepresent disclosure.

DETAILED DESCRIPTION

The following detailed description refers to the accompanying drawings.Wherever possible, the same reference numbers are used in the drawingsand the following description to refer to the same or similar parts.While several illustrative embodiments are described herein,modifications, adaptations and other implementations are possible. Forexample, substitutions, additions, or modifications may be made to thecomponents illustrated in the drawings, and the illustrative methodsdescribed herein may be modified by substituting, reordering, removing,or adding steps to the disclosed methods. Accordingly, the followingdetailed description is not limited to the disclosed embodiments andexamples instead, the proper scope is defined by the appended claims.

The present disclosure is directed to systems and methods for processingimages captured in a retail store. As used herein, the term “retailstore” or simply “store” refers to an establishment offering productsfor sale by direct selection by customers physically or virtuallyshopping within the establishment. The retail store may be anestablishment operated by a single retailer (e.g., supermarket) or anestablishment that includes stores operated by multiple retailers (e.g.,a shopping mall). Embodiments of the present disclosure includereceiving an image depicting a store shelf having at least one productdisplayed thereon. As used herein, the term “store shelf” or simply“shell” refers to any suitable physical structure which may be used fordisplaying products in a retail environment. In one embodiment the storeshelf may be part of a shelving unit including a number of individualstore shelves. In another embodiment, the store shelf may include adisplay unit having a single-level or multi-level surfaces.

Consistent with the present disclosure, the system may process imagesand image data acquired by a capturing device to determine informationassociated with products displayed in the retail store. The term“capturing device” refers to any device configured to acquire image datarepresentative of products displayed in the retail store. Examples ofcapturing devices may include a digital camera, a time-of-flight camera,a stereo camera, an active stereo camera, a depth camera, a Lidarsystem, a laser scanner, CCD based devices, or any other sensor basedsystem capable of converting received light into electric signals. Theterm “image data” refers to any form of data generated based on opticalsignals in the near-infrared, infrared, visible, and ultravioletspectrums (or any other suitable radiation frequency range). Consistentwith the present disclosure, the image data may include pixel datastreams, digital images, digital video streams, data derived fromcaptured images, and data that may be used to construct a 3D image. Theimage data acquired by a capturing device may be transmitted by wired orwireless transmission to a remote server. In one embodiment, thecapturing device may include a stationary camera with communicationlayers (e.g., a dedicated camera fixed to a store shelf, a securitycamera, etc.). Such an embodiment is described in greater detail belowwith reference to FIG. 4A. In another embodiment, the capturing devicemay include a handheld device (e.g., a smartphone, a tablet, a mobilestation, a personal digital assistant, a laptop, and more) or a wearabledevice (e.g., smart glasses, a smartwatch, a clip-on camera). Such anembodiment is described in greater detail below with reference to FIG.4B. In another embodiment, the capturing device may include a roboticdevice with one or more cameras operated remotely or autonomously (e.g.,an autonomous robotic device, a drone, a robot on a track, and more).Such an embodiment is described in greater detail below with referenceto FIG. 4C.

In some embodiments, the capturing device may include one or more imagesensors. The term “image sensor” refers to a device capable of detectingand converting optical signals in the near-infrared, infrared, visible,and ultraviolet spectrums into electrical signals. The electricalsignals may be used to form image data (e,g., an image or a videostream) based on the detected signal. Examples of image sensors mayinclude semiconductor charge-coupled devices (CCD), active pixel sensorsin complementary metal-oxide-semiconductor (CMOS), or N-typemetal-oxide-semiconductors (NMOS, Live MOS). In some cases, the imagesensor may be part of a camera included in the capturing device.

Embodiments of the present disclosure further include analyzing imagesto detect and identify different products. As used herein, the term“detecting a product” may broadly refer to determining an existence ofthe product. For example, the system may determine the existence of aplurality of distinct products displayed on a store shelf. By detectingthe plurality of products, the system may acquire different detailsrelative to the plurality of products (e.g., how many products on astore shelf are associated with a same product type), but it does notnecessarily gain knowledge of the type of product. In contrast, the term“identifying a product” may refer to determining a unique identifierassociated with a specific type of product that allows inventorymanagers to uniquely refer to each product type in a product catalogue.Additionally or alternatively, the term “identifying a product” mayrefer to determining a unique identifier associated with a specificbrand of products that allows inventory managers to uniquely refer toproducts, e.g., based on a specific brand in a product catalogue.Additionally or alternatively, the term “identifying a product” mayrefer to determining a unique identifier associated with a specificcategory of products that allows inventory managers to uniquely refer toproducts, e.g., based on a specific category in a product catalogue. Insome embodiments, the identification may be made based at least in parton visual characteristics of the product (e.g,, size, shape, logo, text,color, etc.). The unique identifier may include any codes that may beused to search a catalog, such as a series of digits, letters, symbols,or any combinations of digits, letters, and symbols. Consistent with thepresent disclosure, the terms “determining a type of a product” and“determining a product type” may also be used interchangeably in thisdisclosure with reference to the term “identifying a product.”

Embodiments of the present disclosure further include determining atleast one characteristic of the product for determining the type of theproduct. As used herein, the term “characteristic of the product” refersto one or more visually discernable features attributed to the product.Consistent with the present disclosure, the characteristic of theproduct may assist in classifying and identifying the product. Forexample, the characteristic of the product may be associated with theornamental design of the product, the size of the product, the shape ofthe product, the colors of the product, the brand of the product, a logoor text associated with the product (e.g., on a product label), andmore. In addition, embodiments of the present disclosure further includedetermining a confidence level associated with the determined type ofthe product. The term “confidence level” refers to any indication,numeric or otherwise, of a level (e.g., within a predetermined range)indicative of an amount of confidence the system has that the determinedtype of the product is the actual type of the product. For example, theconfidence level may have a value between 1 and 10, alternatively, theconfidence level may be expressed as a percentage.

In some cases, the system may compare the confidence level to athreshold. The term “threshold” as used herein denotes a referencevalue, a level, a point, or a range of values, for which, when theconfidence level is above it (or below it depending on a particular usecase), the system may follow a first course of action and, when theconfidence level is below it (or above it depending on a particular usecase), the system may follow a second course of action. The value of thethreshold may be predetermined for each type of product or may bedynamically selected based on different considerations. In oneembodiment, when the confidence level associated with a certain productis below a threshold, the system may obtain contextual information toincrease the confidence level. As used herein, the term “contextualinformation” (or “context”) refers to any information having a direct orindirect relationship with a product displayed on a store shelf. In someembodiments, the system may retrieve different types of contextualinformation from captured image data and/or from other data sources. Insome cases, contextual information may include recognized types ofproducts adjacent to the product under examination. In other cases,contextual information may include text appearing on the product,especially where that text may be recognized (e.g., via OCR) andassociated with a particular meaning. Other examples of types ofcontextual information may include logos appearing on the product, alocation of the product in the retail store, a brand name of theproduct, a price of the product, product information collected frommultiple retail stores, product information retrieved from a catalogassociated with a retail store, etc.

Reference is now made to FIG. 1, which shows an example of a system 100for analyzing information collected from retail stores 105 (for example,retail store 105A, retail store 105B, and retail store 105C). In oneembodiment, system 100 may represent a computer-based system that mayinclude computer system components, desktop computers, workstations,tablets, handheld computing devices, memory devices, and/or internalnetwork(s) connecting the components. System 100 may include or beconnected to various network computing resources (e.g., servers,routers, switches, network connections, storage devices, etc.) necessaryto support the services provided by system 100. In one embodiment,system 100 may enable identification of products in retail stores 105based on analysis of captured images. In another embodiment, system 100may enable a supply of information based on analysis of captured imagesto a market research entity 110 and to different suppliers 115 of theidentified products in retail stores 105 (for example, supplier 115A,supplier 115B, and supplier 115C). In another embodiment, system 100 maycommunicate with a user 120 (sometimes referred to herein as a customer,but which may include individuals associated with a retail environmentother than customers, such as store employee, data collection agent,etc.) about different products in retail stores 105. In one example,system 100 may receive images of products captured by user 120. Inanother example, system 100 may provide to user 120 informationdetermined based on automatic machine analysis of images captured by oneor more capturing devices 125 associated with retail stores 105.

System 100 may also include an image processing unit 130 to execute theanalysis of images captured by the one or more capturing devices 125,image processing unit 130 may include a server 135 operatively connectedto a database 140. Image processing unit 130 may include one or moreservers connected by a communication network, a cloud platform, and soforth. Consistent with the present disclosure, image processing unit 130may receive raw or processed data from capturing device 125 viarespective communication links, and provide information to differentsystem components using a network 150. Specifically, image processingunit 130 may use any suitable image analysis technique including, forexample, object recognition, object detection, image segmentation,feature extraction, optical character recognition (OCR), object-basedimage analysis, shape region techniques, edge detection techniques,pixel-based detection, artificial neural networks, convolutional neuralnetworks, etc. In addition, image processing unit 130 may useclassification algorithms to distinguish between the different productsin the retail store. In some embodiments, image processing unit 130 mayutilize suitably trained machine learning algorithms and models toperform the product identification. Network 150 may facilitatecommunications and data exchange between different system componentswhen these components are coupled to network 150 to enable output ofdata derived from the images captured by the one or more capturingdevices 125. In some examples, the types of outputs that imageprocessing unit 130 can generate may include identification of products,indicators of product quantity, indicators of planogram compliance,indicators of service-improvement events (e.g., a cleaning event, arestocking event, a rearrangement event, etc.), and various reportsindicative of the performances of retail stores 105. Additional examplesof the different outputs enabled by image processing unit 130 aredescribed below with reference to FIGS. 11A-11E and throughout thedisclosure.

Consistent with the present disclosure, network 150 may be any type ofnetwork (including infrastructure) that provides communications,exchanges information, and/or facilitates the exchange of informationbetween the components of system 100. For example, network 150 mayinclude or be part of the Internet, a Local Area Network, wirelessnetwork (e.g., a Wi-Fi/302.11 network), or other suitable connections.In other embodiments, one or more components of system 100 maycommunicate directly through dedicated communication links, such as, forexample, a telephone network, an extranet, an intranet, the Internet,satellite communications, off-line communications, wirelesscommunications, transponder communications, a local area network (LAN),a wide area network (WAN), a virtual private network (VPN), and soforth.

In one example configuration, server 135 may be a cloud server thatprocesses images received directly (or indirectly) from one or morecapturing device 125 and processes the images to detect and/or identifyat least some of the plurality of products in the image based on visualcharacteristics of the plurality of products. The term “cloud server”refers to a computer platform that provides services via a network, suchas the Internet. In this example configuration, server 135 may usevirtual machines that may not correspond to individual hardware. Forexample, computational and/or storage capabilities may be implemented byallocating appropriate portions of desirable computation/storage powerfrom a scalable repository, such as a data center or a distributedcomputing environment. In one example, server 135 may implement themethods described herein using customized hard-wired logic, one or moreApplication Specific Integrated Circuits (ASICs) or Field ProgrammableGate Arrays (FPGAs), firmware, and/or program logic which, incombination with the computer system, cause server 135 to be aspecial-purpose machine.

In another example configuration, server 135 may be part of a systemassociated with a retail store that communicates with capturing device125 using a wireless local area network (WLAN) and may provide similarfunctionality as a cloud server. In this example configuration, server135 may communicate with an associated cloud server (not shown) andcloud database (not shown). The communications between the store serverand the cloud server may be used in a quality enforcement process, forupgrading the recognition engine and the software from time to time, forextracting information from the store level to other data users, and soforth. Consistent with another embodiment, the communications betweenthe store server and the cloud server may be discontinuous (purposely orunintentional) and the store server may be configured to operateindependently from the cloud server. For example, the store server maybe configured to generate a record indicative of changes in productplacement that occurred when there was a limited connection (or noconnection) between the store server and the cloud server, and toforward the record to the cloud server once connection is reestablished.

As depicted in FIG. 1, server 135 may be coupled to one or more physicalor virtual storage devices such as database 140. Server 135 may accessdatabase 140 to detect and/or identify products. The detection may occurthrough analysis of features in the image using an algorithm and storeddata. The identification may occur through analysis of product featuresin the image according to stored product models. Consistent with thepresent embodiment, the term “product model” refers to any type ofalgorithm or stored product data that a processor may access or executeto enable the identification of a particular product associated with theproduct model. For example, the product model may include a descriptionof visual and contextual properties of the particular product (e.g., theshape, the size, the colors, the texture, the brand name, the price, thelogo, text appearing on the particular product, the shelf associatedwith the particular product, adjacent products in a planogram, thelocation within the retail store, etc.). In some embodiments, a singleproduct model may be used by server 135 to identify more than one typeof products, such as, when two or more product models are used incombination to enable identification of a product. For example, in somecases, a first product model may be used by server 135 to identify aproduct category (such models may apply to multiple product types, e.g.,shampoo, soft drinks, etc.), and a second product model may be used byserver 135 to identify the product type, product identity, or othercharacteristics associated with a product. In some cases, such productmodels may be applied together (e.g., in series, in parallel, in acascade fashion, in a decision tree fashion, etc.) to reach a productidentification. In other embodiments, a single product model may be usedby server 135 to identify a particular product type (e.g., 6-pack of 16oz Coca-Cola Zero).

Database 140 may be included on a volatile or non-volatile, magnetic,semiconductor, tape, optical, removable, non-removable, or other type ofstorage device or tangible or non-transitory computer-readable medium.Database 140 may also be part of server 135 or separate from server 135.When database 140 is not part of server 135, server 135 may exchangedata with database 140 via a communication link. Database 140 mayinclude one or more memory devices that store data and instructions usedto perform one or more features of the disclosed embodiments. In oneembodiment, database 140 may include any suitable databases, rangingfrom small databases hosted on a work station to large databasesdistributed among data centers. Database 140 may also include anycombination of one or more databases controlled by memory controllerdevices (e.g., server(s), etc.) or software. For example, database 140may include document management systems, Microsoft SQL databases, SharePoint databases, Oracle™ databases, Sybase™ databases, other relationaldatabases, or non-relational databases, such as mango and others.

Consistent with the present disclosure, image processing unit 130 maycommunicate with output devices 145 to present information derived basedon processing of image data acquired by capturing devices 125. The term“output device” is intended to include all possible types of devicescapable of outputting information from server 135 to users or othercomputer systems (e.g., a display screen, a speaker, a desktop computer,a laptop computer, mobile device, tablet, a PDA, etc.), such as 145A,145B, 1450 and 145D. In one embodiment each of the different systemcomponents (i.e., retail stores 105, market research entity 110,suppliers 115, and users 120) may be associated with an output device145, and each system component may be configured to present differentinformation on the output device 145. In one example, server 135 mayanalyze acquired images including representations of shelf spaces. Basedon this analysis, server 135 may compare shelf spaces associated withdifferent products, and output device 145A may present market researchentity 110 with information about the shelf spaces associated withdifferent products. The shelf spaces may also be compared with salesdata, expired products data, and more. Consistent with the presentdisclosure, market research entity 110 may be a part of (or may workwith) supplier 115. In another example, server 135 may determine productcompliance to a predetermined planogram, and output device 145B maypresent to supplier 115 information about the level of productcompliance at one or more retail stores 105 (for example in a specificretail store 105, in a group of retail stores 105 associated withsupplier 115, in all retail stores 105, and so forth). The predeterminedplanogram may be associated with contractual obligations and/or otherpreferences related to the retailer methodology for placement ofproducts on the store shelves. In another example, server 135 maydetermine that a specific store shelf has a type of fault in the productplacement, and output device 145C may present to a manager of retailstore 105 a user-notification that may include information about acorrect display location of a misplaced product, information about astore shelf associated with the misplaced product, information about atype of the misplaced product, and/or a visual depiction of themisplaced product. In another example, server 135 may identify whichproducts are available on the shelf and output device 145D may presentto user 120 an updated list of products.

The components and arrangements shown in FIG. 1 are not intended tolimit the disclosed embodiments, as the system components used toimplement the disclosed processes and features may vary. In oneembodiment, system 100 may include multiple servers 135, and each server135 may host a certain type of service. For example, a first server mayprocess images received from capturing devices 125 to identify at leastsome of the plurality of products in the image, and a second server maydetermine from the identified products in retail stores 105 compliancewith contractual obligations between retail stores 105 and suppliers115. In another embodiment, system 100 may include multiple servers 135,a first type of servers 135 that may process information from specificcapturing devices 125 (e.g., handheld devices of data collection agents)or from specific retail stores 105 (e.g., a server dedicated to aspecific retail store 105 may be placed in or near the store). System100 may further include a second type of servers 135 that collect andprocess information from the first type of servers 135.

FIG. 2 is a block diagram representative of an example configuration ofserver 135. In one embodiment, server 135 may include a bus 200 (or anyother communication mechanism) that interconnects subsystems andcomponents for transferring information within server 135. For example,bus 200 may interconnect a processing device 202, a memory interface204, a network interface 206, and a peripherals interface 208 connectedto an I/O system 210.

Processing device 202, shown in FIG. 2, may include at least oneprocessor configured to execute computer programs, applications,methods, processes, or other software to execute particular instructionsassociated with embodiments described in the present disclosure. Theterm “processing device” refers to any physical device having anelectric circuit that performs a logic operation. For example,processing device 202 may include one or more processors, integratedcircuits, microchips, microcontrollers, microprocessors, all or part ofa central processing unit (CPU), graphics processing unit (GPU), digitalsignal processor (DSP), field programmable gate array (FPGA), or othercircuits suitable for executing instructions or performing logicoperations. Processing device 202 may include at least one processorconfigured to perform functions of the disclosed methods such as amicroprocessor manufactured by Intel™, Nvidia™, manufactured by AMD™,and so forth. Processing device 202 may include a single core ormultiple core processors executing parallel processes simultaneously. Inone example, processing device 202 may be a single core processorconfigured with virtual processing technologies. Processing device 202may implement virtual machine technologies or other technologies toprovide the ability to execute, control, run, manipulate, store, etc,,multiple software processes, applications, programs, etc. In anotherexample, processing device 202 may include a multiple-core processorarrangement (e.g., dual, quad core, etc.) configured to provide parallelprocessing functionalities to allow a device associated with processingdevice 202 to execute multiple processes simultaneously. It isappreciated that other types of processor arrangements could beimplemented to provide the capabilities disclosed herein.

Consistent with the present disclosure, the methods and processesdisclosed herein may be performed by server 135 as a result ofprocessing device 202 executing one or more sequences of one or moreinstructions contained in a non-transitory computer-readable storagemedium. As used herein, a non-transitory computer-readable storagemedium refers to any type of physical memory on which information ordata readable by at least one processor can be stored. Examples includerandom access memory (RAM), read-only memory (ROM), volatile memory,nonvolatile memory, hard drives, CD ROMs, DVDs, flash drives, disks, anyother optical data storage medium, any physical medium with patterns ofholes, a RAM, a PROM, an EPROM, a FLASH-EPROM or any other flash memory,NVRAM, a cache, a register, any other memory chip or cartridge, andnetworked versions of the same. The terms “memory” and“computer-readable storage medium” may refer to multiple structures,such as a plurality of memories or computer-readable storage mediumslocated within server 135, or at a remote location. Additionally, one ormore computer-readable storage mediums can be utilized in implementing acomputer-implemented method. The term “computer-readable storage medium”should be understood to include tangible items and exclude carrier wavesand transient signals.

According to one embodiment, server 135 may include network interface206 (which may also be any communications interface) coupled to bus 200.Network interface 206 may provide one-way or two-way data communicationto a local network, such as network 150. Network interface 206 mayinclude an integrated services digital network (ISDN) card, cablemodern, satellite modem, or a modem to provide a data communicationconnection to a corresponding type of telephone line. As anotherexample, network interface 206 may include a local area network (LAN)card to provide a data communication connection to a compatible LAN. Inanother embodiment, network interface 206 may include an Ethernet portconnected to radio frequency receivers and transmitters and/or optical(e.g., infrared) receivers and transmitters. The specific design andimplementation of network interface 206 depends on the communicationsnetwork(s) over which server 135 is intended to operate. As describedabove, server 135 may be a cloud server or a local server associatedwith retail store 105. In any such implementation, network interface 206may be configured to send and receive electrical, electromagnetic, oroptical signals, through wires or wirelessly, that may carry analog ordigital data streams representing various types of information. Inanother example, the implementation of network interface 206 may besimilar or identical to the implementation described below for networkinterface 306.

Server 135 may also include peripherals interface 208 coupled to bus200. Peripherals interface 208 may be connected to sensors, devices, andsubsystems to facilitate multiple functionalities. In one embodiment,peripherals interface 208 may be connected to I/O system 210 configuredto receive signals or input from devices and provide signals or outputto one or more devices that allow data to he received and/or transmittedby server 135. In one embodiment system 210 may include or be associatedwith output device 145. For example, I/O system 210 may include a touchscreen controller 212, an audio controller 214, and/or other inputcontroller(s) 216. Touch screen controller 212 may he coupled to a touchscreen 218. Touch screen 218 and touch screen controller 212 can, forexample, detect contact, movement, or break thereof using any of aplurality of touch sensitivity technologies, including but not limitedto capacitive, resistive, infrared, and surface acoustic wavetechnologies as well as other proximity sensor arrays or other elementsfor determining one or more points of contact with touch screen 218.Touch screen 218 may also, for example, be used to implement virtual orsoft buttons and/or a keyboard. In addition to or instead of touchscreen 218, I/O system 210 may include a display screen (e.g., CRT, LCD,etc.), virtual reality device, augmented reality device, and so forth.Specifically, touch screen controller 212 (or display screen controller)and touch screen 218 (or any of the alternatives mentioned above) mayfacilitate visual output from server 135. Audio controller 214 may becoupled to a microphone 220 and a speaker 222 to facilitatevoice-enabled functions, such as voice recognition, voice replication,digital recording, and telephony functions. Specifically, audiocontroller 214 and speaker 222 may facilitate audio output from server135. The other input controller(s) 216 may be coupled to otherinput/control devices 224, such as one or more buttons, keyboards,rocker switches, thumb-wheel, infrared port, USB port, image sensors,motion sensors, depth sensors, and/or a pointer device such as acomputer mouse or a stylus.

In some embodiments, processing device 202 may use memory interface 204to access data and a software product stored on a memory device 226.Memory device 226 may include operating system programs for server 135that perform operating system functions when executed by the processingdevice. By way of example, the operating system programs may includeMicrosoft Windowsr™, Unix™, Linux™, Apple™ operating systems, personaldigital assistant (PDA) type operating systems such as Apple iOS, GoogleAndroid, Blackberry OS, or other types of operating systems.

Memory device 226 may also store communication instructions 223 tofacilitate communicating with one or more additional devices (e.g.,capturing device 125), one or more computers (e.g., output devices145A-145D) and/or one or more servers. Memory device 226 may includegraphical user interface instructions 230 to facilitate graphic userinterface processing; image processing instructions 232 to facilitateimage data processing-related processes and functions; sensor processinginstructions 234 to facilitate sensor-related processing and functions;web browsing instructions 236 to facilitate web browsing-relatedprocesses and functions; and other software instructions 238 tofacilitate other processes and functions. Each of the above identifiedinstructions and applications may correspond to a set of instructionsfor performing one or more functions described above. These instructionsneed not he implemented as separate software programs, procedures, ormodules. Memory device 226 may include additional instructions or fewerinstructions. Furthermore, various functions of server 135 may beimplemented in hardware and/or in software, including in one or moresignal processing and/or application specific integrated circuits. Forexample, server 135 may execute an image processing algorithm toidentify in received images one or more products and/or obstacles, suchas shopping carts, people, and more.

In one embodiment, memory device 226 may store database 140. Database140 may include product type model data 240 (e.g., an imagerepresentation, a list of features, a model obtained by training machinelearning algorithm using training examples, an artificial neuralnetwork, and more) that may be used to identify products in receivedimages; contract-related data 242 (e.g., planograms, promotions data,etc.) that may be used to determine if the placement of products on thestore shelves and/or the promotion execution are consistent withobligations of retail store 105; catalog data 244 (e.g., retail storechain's catalog, retail store's master file, etc.) that may be used tocheck if all product types that should be offered in retail store 105are in fact in the store, if the correct price is displayed next to anidentified product, etc.; inventory data 246 that may be used todetermine if additional products should be ordered from suppliers 115;employee data 248 (e.g., attendance data, records of training provided,evaluation and other performance-related communications, productivityinformation, etc.) that may be used to assign specific employees tocertain tasks; and calendar data 250 (e.g., holidays, national days,international events, etc.) that may be used to determine if a possiblechange in a product model is associated with a certain event. In otherembodiments of the disclosure, database 140 may store additional typesof data or fewer types of data. Furthermore, various types of data maybe stored in one or more memory devices other than memory device 226.

The components and arrangements shown in FIG. 2 are not intended tolimit the disclosed embodiments. As will be appreciated by a personskilled in the art having the benefit of this disclosure, numerousvariations and/or modifications may be made to the depictedconfiguration of server 135. For example, not all components may beessential for the operation of server 135 in all cases. Any componentmay be located in any appropriate part of server 135, and the componentsmay be rearranged into a variety of configurations while providing thefunctionality of the disclosed embodiments. For example, some serversmay not include some of the elements shown in 110 system 215.

FIG. 3 is a block diagram representation of an example configuration ofcapturing device 125. In one embodiment, capturing device 125 mayinclude a processing device 302, a memory interface 304, a networkinterface 306, and a peripherals interface 308 connected to image sensor310. These components can be separated or can be integrated in one ormore integrated circuits. The various components in capturing device 125can be coupled by one or more communication buses or signal lines (e,g.,bus 300). Different aspects of the functionalities of the variouscomponents in capturing device 125 may be understood from thedescription above regarding components of server 135 having similarfunctionality.

According to one embodiment, network interface 306 may be used tofacilitate communication with server 135. Network interface 306 may bean Ethernet port connected to radio frequency receivers and transmittersand/or optical receivers and transmitters. The specific design andimplementation of network interface 306 depends on the communicationsnetwork(s) over which capturing device 125 is intended to operate. Forexample, in some embodiments, capturing device 125 may include a networkinterface 306 designed to operate over a GSM network, a CPRS network, anEDGE network, a Wi-Fi WiMax network, a Bluetooth® network, etc. Inanother example, the implementation of network interface 306 may besimilar or identical to the implementation described above for networkinterface 206.

In the example illustrated in FIG. 3, peripherals interface 308 ofcapturing device 125 may be connected to at least one image sensor 310associated with at least one lens 312 for capturing image data in anassociated field of view. In some configurations, capturing device 125may include a plurality of image sensors associated with a plurality oflenses 312. In other configurations, image sensor 310 may be part of acamera included in capturing device 125. According to some embodiments,peripherals interface 308 may also be connected to other sensors (notshown), such as a motion sensor, a light sensor, infrared sensor, soundsensor, a proximity sensor, a temperature sensor, a biometric sensor, orother sensing devices to facilitate related functionalities. Inaddition, a positioning sensor may also be integrated with, or connectedto, capturing device 125. For example, such positioning sensor may beimplemented using one of the following technologies: Global PositioningSystem (GPS), GLobal NAvigation Satellite System (GLONASS), Galileoglobal navigation system, BeiDou navigation system, other GlobalNavigation Satellite Systems (GNSS), Indian Regional NavigationSatellite System (IRNSS), Local Positioning Systems (LPS), Real-TimeLocation Systems (RTLS), Indoor Positioning System (IPS), Wi-Fi basedpositioning systems, cellular triangulation, and so forth. For example,the positioning sensor may be built into mobile capturing device 125,such as smartphone devices. In another example, position software mayallow mobile capturing devices to use internal or external positioningsensors e,g., connecting via a serial port or Bluetooth).

Consistent with the present disclosure, capturing device 125 may includedigital components that collect data from image sensor 310, transform itinto an image, arid store the image on a memory device 314 and/ortransmit the image using network interface 306. In one embodiment,capturing device 125 may be fixedly mountable to a store shelf or toother objects in the retail store (such as walls, ceilings, floors,refrigerators, checkout stations, displays, dispensers, rods which maybe connected to other objects in the retail store, and so forth). In oneembodiment, capturing device 125 may be split into at least two housingssuch that only image sensor 310 and lens 312 may be visible on the storeshelf, and the rest of the digital components may be located in aseparate housing. An example of this type of capturing device isdescribed below with reference to FIGS. 5-7.

Consistent with the present disclosure, capturing device 125 may usememory interface 304 to access memory device 314. Memory device 314 mayinclude high-speed, random access memory and/or non-volatile memory suchas one or more magnetic disk storage devices, one or more opticalstorage devices, and/or flash memory (e.g., NAND, NOR) to store capturedimage data. Memory device 314 may store operating system instructions316, such as DARWIN, RTXC, LINUX, IOS, UNIX, LINUX, OS X, WINDOWS, or anembedded operating system such as VXWorkS. Operating system 316 caninclude instructions for handling basic system services and forperforming hardware dependent tasks. In some implementations, operatingsystem 316 may include a kernel (e.g., UNIX kernel, LINUX kernel, etc.).In addition, memory device 314 may store capturing instructions 318 tofacilitate processes and functions related to image sensor 310;graphical user interface instructions 320 that enables a user associatedwith capturing device 125 to control the capturing device and/or toacquire images of an area-of-interest in a retail establishment; andapplication instructions 322 to facilitate a process for monitoringcompliance of product placement or other processes.

The components and arrangements shown in FIG. 3 are not intended tolimit the disclosed embodiments. As will be appreciated by a personskilled in the art having the benefit of this disclosure, numerousvariations and/or modifications may be made to the depictedconfiguration of capturing device 125. For example, not all componentsare essential for the operation of capturing device 125 in all cases.Any component may be located in any appropriate part of capturing device125, and the components may be rearranged into a variety ofconfigurations while providing the functionality of the disclosedembodiments. For example, some capturing devices may not have lenses,and other capturing devices may include an external memory deviceinstead of memory device 314.

FIGS. 4A-4C illustrate example configurations for capturing image datain retail store 105 according to disclosed embodiments. FIG. 4Aillustrates how an aisle 400 of retail store 105 may be imaged using aplurality of capturing devices 125 fixedly connected to store shelves.FIG. 4B illustrates how aisle 400 of retail store 105 may be imagedusing a handheld communication device. FIG. 4C illustrates how aisle 400of retail store 105 may be imaged by robotic devices equipped withcameras.

With reference to FIG. 4A and consistent with the present disclosure,retail store 105 may include a plurality of capturing devices 125fixedly mounted (for example, to store shelves, walls, ceilings, floors,refrigerators, checkout stations, displays, dispensers, rods which maybe connected to other objects in the retail store, and so forth) andconfigured to collect image data. As depicted, one side of an aisle 400may include a plurality of capturing devices 125 (e.g., 125A, 125B, and125C) fixedly mounted thereon and directed such that they may captureimages of an opposing side of aisle 400. The plurality of capturingdevices 125 may be connected to an associated mobile power source (e.g.,one or more batteries), to an external power supply (e.g., a powergrid), obtain electrical power from a wireless power transmissionsystem, and so forth. As depicted in FIG. 4A, the plurality of capturingdevices 125 may be placed at different heights and at least theirvertical fields of view may be adjustable. Generally, both sides ofaisle 400 may include capturing devices 125 in order to cover both sidesof aisle 400.

Differing numbers of capturing devices 125 may be used to cover shelvingunit 402. In addition, there may be an overlap region in the horizontalfield of views of some of capturing devices 125. For example, thehorizontal fields of view of capturing devices (e.g., adjacent capturingdevices) may at least partially overlap with one another. In anotherexample, one capturing device may have a lower field of view than thefield of view of a second capturing device, and the two capturingdevices may have at least partially overlapping fields of view.According to one embodiment, each capturing device 125 may be equippedwith network interface 306 for communicating with server 135. In oneembodiment, the plurality of capturing devices 125 in retail store 105may be connected to server 135 via a single WLAN. Network interface 306may transmit information associated with a plurality of images capturedby the plurality of capturing devices 125 for analysis purposes. In oneexample, server 135 may determine an existence of an occlusion event(such as, by a person, by store equipment, such as a ladder, cart, etc.)and may provide a notification to resolve the occlusion event. Inanother example, server 135 may determine if a disparity exists betweenat least one contractual obligation and product placement as determinedbased on automatic analysis of the plurality of images. The transmittedinformation may include raw images, cropped images, processed imagedata, data about products identified in the images, and so forth.Network interface 306 may also transmit information identifying thelocation of the plurality capturing devices 125 in retail store 105.

With reference to FIG. 4B and consistent with the present disclosure,server 135 may receive image data captured by users 120. In a firstembodiment, server 135 may receive image data acquired by storeemployees. In one implementation, a handheld device of a store employee(e.g., capturing device 125D) may display a real-time video streamcaptured by the image sensor of the handheld device. The real-time videostream may be augmented with markings identifying to the store employeean area-of-interest that needs manual capturing of images. One of thesituations in which manual image capture may be desirable may occurwhere the area-of-interest is outside the fields of view of a pluralityof cameras fixedly connected to store shelves in aisle 400. In othersituations, manual capturing of images of an area-of-interest may bedesirable when a current set of acquired images is out of date (e.g.,obsolete in at least one respect) or of poor quality (e.g., lackingfocus, obstacles, lesser resolution, lack of light, etc.). Additionaldetails of this embodiment are described in Applicant's InternationalPatent Application No, PCT/TB2018/001107, which is incorporated hereinby reference,

In a second embodiment, server 135 may receive image data acquired bycrowd sourcing. In one exemplary implementation, server 135 may providea request to a detected mobile device for an updated image of thearea-of-interest in aisle 400. The request may include an incentive(e.g., $2 discount) to user 120 for acquiring the image. In response tothe request, user 120 may acquire and transmit an up-to-date image ofthe area-of-interest. After receiving the image from user 120, server135 may transmit the accepted incentive or agreed upon reward to user120. The incentive may comprise a text notification and a redeemablecoupon. In some embodiments, the incentive may include a redeemablecoupon for a product associated with the area-of-interest. Server 135may generate image-related data based on aggregation of data from imagesreceived from crowd sourcing and from images received from a pluralityof cameras fixedly connected to store shelves. Additional details ofthis embodiment are described in Applicant's International PatentApplication No. PCT/TB2017/000919, which is incorporated herein byreference.

With reference to FIG. 4C and consistent with the present disclosure,server 135 may receive image data captured by robotic devices withcameras traversing in aisle 400. The present disclosure is not limitedto the type of robotic devices used to capture images of retail store105. In some embodiments, the robotic devices may include a robot on atrack (e.g., a Cartesian robot configured to move along an edge of ashelf or in parallel to a shelf, such as capturing device 125E), a drone(e.g., capturing device 125F), and/or a robot that may move on the floorof the retail store (e.g., a wheeled robot such as capturing device125G, a legged robot, a snake-like robot, etc,). The robotic devices maybe controlled by server 135 and may be operated remotely orautonomously. In one example, server 135 may instruct capturing device125E to perform periodic scans at times when no customers or otherobstructions are identified in aisle 400. Specifically, capturing device125E may be configured to move along store shelf 404 and to captureimages of products placed on store shelf 404, products placed on storeshelf 406, or products located on shelves opposite store shelf (e.g.,store shelf 408). In another example, server 135 may instruct capturingdevice 125F to perform a scan of all the area of retail store 105 beforethe opening hour. In another example, server 135 may instruct capturingdevice 125G to capture a specific area-of-interest, similar as describedabove with reference to receiving images acquired by the storeemployees. In some embodiments, robotic capturing devices (such as 125Fand 125G) may include an internal processing unit that may allow them tonavigate autonomously within retail store 105. For example, the roboticcapturing devices may use input from sensors (e.g., image sensors, depthsensors, proximity sensors, etc.), to avoid collision with objects orpeople, and to complete the scan of the desired area of retail store105.

As discussed above with reference to FIG. 4A, the image datarepresentative of products displayed on store shelves may be acquired bya plurality of stationary capturing devices 125 fixedly mounted in theretail store. One advantage of having stationary image capturing devicesspread throughout retail store 105 is the potential for acquiringproduct images from set locations and on an ongoing basis such thatup-to-date product status may be determined for products throughout aretail store at any desired periodicity (e.g., in contrast to a movingcamera system that may acquire product images more infrequently).However, there may be certain challenges in this approach. The distancesand angles of the image capturing devices relative to the capturedproducts should be selected such as to enable adequate productidentification, especially when considered in view of image sensorresolution and/or optics specifications. For example, a capturing deviceplaced on the ceiling of retail store 105 may have sufficientresolutions and optics to enable identification of large products (e.g.,a pack of toilet paper), but may be insufficient for identifying smallerproducts (e.g., deodorant packages). The image capturing devices shouldnot occupy shelf space that is reserved for products for sale. The imagecapturing devices should not be positioned in places where there is alikelihood that their fields of view will be regularly blocked bydifferent objects. The image capturing devices should be able tofunction for long periods of time with minimum maintenance. For example,a requirement for frequent replacement of batteries may render certainimage acquisition systems cumbersome to use, especially where many imageacquisition devices are in use throughout multiple locations in a retailstore and across multiple retail stores. The image capturing devicesshould also include processing capabilities and transmissioncapabilities for providing real time or near real time image data aboutproducts. The disclosed image acquisition systems address thesechallenges.

FIG. 5A illustrates an example of a system 500 for acquiring images ofproducts in retail store 105. Throughout the disclosure, capturingdevice 125 may refer to a system, such as system 500 shown in FIG. 5A.As shown, system 500 may include a first housing 502 configured forlocation on a retail shelving unit (e.g., as illustrated in FIG. 5B),and a second housing 504 configured for location on the retail shelvingunit separate from first housing 502. The first and the second housingmay be configured for mounting on the retail shelving unit in anysuitable way (e.g., screws, bolts, clamps, adhesives, magnets,mechanical means, chemical means, etc.). In some embodiments, firsthousing 502 may include an image capture device 506 (e.g., a cameramodule that may include image sensor 310) and second housing 504 mayinclude at least one processor (e.g., processing device 302) configuredto control image capture device 506 and also to control a networkinterface (e.g., network interface 306) for communicating with a remoteserver e.g., server 135).

System 500 may also include a data conduit 508 extending between firsthousing 502 and second housing 504. Data conduit 508 may be configuredto enable transfer of control signals from the at least one processor toimage capture device 506 and to enable collection of image data acquiredby image capture device 506 for transmission by the network interface.Consistent with the present disclosure, the term “data conduit” mayrefer to a communications channel that may include either a physicaltransmission medium such as a wire or a logical connection over amultiplexed medium such as a radio channel. In some embodiments, dataconduit 508 may be used for conveying image data from image capturedevice 506 to at least one processor located in second housing 504.Consistent with one implementation of system 500, data conduit 508 mayinclude flexible printed circuits and may have a length of at leastabout 5 cm, at least about 10 cm, at least about 15 cm, etc. The lengthof data conduit 508 may be adjustable to enable placement of firsthousing 502 separately from second housing 504. For example, in someembodiments, data conduit may be retractable within second housing 504such that the length of data conduit exposed between first housing 502and second housing 504 may be selectively adjusted.

In one embodiment, the length of data conduit 508 may enable firsthousing 502 to be mounted on a first side of a horizontal store shelffacing the aisle (e.g., store shelf 510 illustrated in FIG. 5B) andsecond housing 504 to be mounted on a second side of store shelf 510that faces the direction of the ground (e.g., an underside of a storeshelf). In this embodiment, data conduit 508 may be configured to bendaround an edge of store shelf 510 or otherwise adhere/follow contours ofthe shelving unit. For example, a first portion of data conduit 508 maybe configured for location on the first side of store shelf 510 (e.g., aside facing an opposing retail shelving unit across an aisle) and asecond portion of data conduit 508 may be configured for location on asecond side of store shelf 510 (e.g., an underside of the shelf, whichin some cases may be orthogonal to the first side). The second portionof data conduit 508 may be longer than the first portion of data conduit508. Consistent with another embodiment, data conduit 508 may beconfigured for location within an envelope of a store shelf. Forexample, the envelope may include the outer boundaries of a channellocated within a store shelf, a region on an underside of art L-shapedstore shelf, a region between two store shelves, etc. Consistent withanother implementation of system 500 discussed below, data conduit 508may include a virtual conduit associated with a wireless communicationslink between first housing 502 and second housing 504.

FIG. 5B illustrates an exemplary configuration for mounting firsthousing 502 on store shelf 510. Consistent with the present disclosure,first housing 502 may be placed on store shelf 510, next to or embeddedin a plastic cover that may be used for displaying prices.Alternatively, first housing 502 may be placed or mounted on any otherlocation in retail store 105. For example, first housing 502 may beplaced or mounted on the walls, on the ceiling, on refrigerator units,on display units, and more. The location and/or orientation of firsthousing 502 may be selected such that a field of view of image capturedevice 506 may cover at least a portion of an opposing retail shelvingunit. Consistent with the present disclosure, image capture device 506may have a view angle of between 50 and 80 degrees, about 62 degrees,about 67 degrees, or about 75 degrees. Consistent with the presentdisclosure, image capture device 506 may include an image sensor havingsufficient image resolution to enable detection of text associated withlabels on an opposing retail shelving unit. In one embodiment, the imagesensor may include m*n pixels. For example, image capture device 506 mayhave an 8 MP image sensor that includes an array of 3280*2464 pixels.Each pixel may include at least one photo-voltaic cell that converts thephotons of the incident light to an electric signal. The electricalsignal may be converted to digital data by an A/D converter andprocessed by the image processor (ISP). In one embodiment, the imagesensor of image capture device 506 may be associated with a pixel sizeof between 1.1×1.1 um² and 1.7×1.7 um², for example, 1.4×1.4 um².

Consistent with the present disclosure, image capture device 506 may beassociated with a lens (e.g., lens 312) having a fixed focal lengthselected according to a distance expected to be encountered betweenretail shelving units on opposite sides of an aisle (e.g., distance d1shown in FIG. 6A) and/or according to a distance expected to beencountered between a side of a shelving unit facing the aisle on oneside of an aisle and a side of a shelving unit facing away of the aisleon the other side of the aisle (e.g., distance d2 shown in FIG. 6A). Thefocal length may also be based on any other expected distance betweenthe image acquisition device and products to be imaged. As used herein,the term “focal length” refers to the distance from the optical centerof the lens to a point where objects located at the point aresubstantially brought into focus. In contrast to zoom lenses, in fixedlenses the focus is not adjustable. The focus is typically set at thetime of lens design and remains fixed. In one embodiment, the focallength of lens 312 may be selected based on the distance between twosides of aisles in the retail store (e.g,, distance d1, distance d2, andso forth). In some embodiments, image capture device 506 may include alens with a fixed focal length having a fixed value between 2.5 mm and4.5 mm, such as about 3.1 mm, about 3.4 mm, about 3.7 mm. For example,when distance d 1 between two opposing retail shelving units is about 2meters, the focal length of the lens may be about 3.6 mm. Unlessindicated otherwise, the term “about” with regards to a numeric value isdefined as a variance of up to 5% with respect to the stated value. Ofcourse, image capture devices having non-fixed focal lengths may also beused depending on the requirements of certain imaging environments, thepower and space resources available, etc.

FIG. 5C illustrates an exploded view of second housing 504. In someembodiments, the network interface located in second housing 504 (e.g.,network interface 306) may be configured to transmit to server 135information associated with a plurality of images captured by imagecapture device 506. For example, the transmitted information may be usedto determine if a disparity exists between at least one contractualobligation (e.g. planogram) and product placement. In one example, thenetwork interface may support transmission speeds of 0.5 Mb/s, 1 Mb/s, 5MB/s, or more. Consistent with the present disclosure, the networkinterface may allow different modes of operations to be selected, suchas: high-speed, slope-control, or standby. In high-speed mode,associated output drivers may have fast output rise and fall times tosupport high-speed bus rates; in slope-control, the electromagneticinterference may be reduced and the slope (i.e., the change of voltageper unit of time) may be proportional to the current output; and instandby mode, the transmitter may be switched off and the receiver mayoperate at a lower current.

Consistent with the present disclosure, second housing 504 may include apower port 512 for conveying energy from a power source to first housing502. In one embodiment, second housing 504 may include a section for atleast one mobile power source 514 (e.g., in the depicted configurationthe section is configured to house four batteries). The at least onemobile power source may provide sufficient power to enable image capturedevice 506 to acquire more than 1,000 pictures, more than 5,000pictures, more than 10,000 pictures, or more than 15,000 pictures, andto transmit them to server 135. In one embodiment, mobile power source514 located in a single second housing 504 may power two or more imagecapture devices 506 mounted on the store shelf. For example, as depictedin FIGS. 6A and 6B, a single second housing 504 may be connected to aplurality of first housings 502 with a plurality of image capturedevices 506 covering different (overlapping or non-overlapping) fieldsof view.

Accordingly, the two or more image capture devices 506 may be powered bya single mobile power source 514 and/or the data captured by two or moreimage capture devices 506 may be processed to generate a panoramic imageby a single processing device located in second housing 504. In additionto mobile power source 514 or as an alternative to mobile power source514, second housing 504 may also be connected to an external powersource. For example, second housing 504 may be mounted to a store shelfand connected to an electric power grid. In this example, power port 512may be connected to the store shelf through a wire for providingelectrical power to image capture device 506. In another example, aretail shelving unit or retail store 105 may include a wireless powertransmission system, and power port 512 may be connected to a deviceconfigured to obtain electrical power from the wireless powertransmission system. In addition, as discussed below, system 500 may usepower management policies to reduce the power consumption. For example,system 500 may use selective image capturing and/or selectivetransmission of images to reduce the power consumption or conservepower.

FIG. 6A illustrates a schematic diagram of a top view of aisle 600 inretail store 105 with multiple image acquisition systems 500 (e.g.,500A, 500B, 500C, 5000, and 500E) deployed thereon for acquiring imagesof products. Aisle 600 may include a first retail shelving unit 602 anda second retail shelving unit 604 that opposes first retail shelvingunit 602. In some embodiments, different numbers of systems 500 may bemounted on opposing retail shelving units. For example, system 500A(including first housing 502A, second housing 504A, and data conduit508A), system 500B (including first housing 502B second housing 50413,and data conduit 508B), and system 500C (including first housing 502C,second housing 504C, and data conduit 508C) may be mounted on firstretail shelving unit 602; and system 500D (including first housing502D1, first housing 502D2, second housing 504D, and data conduits 508D1and 508D2) and system 500E (including first housing 502E1, first housing502E2, second housing 504E, and data conduits 508E1 and 508E2) may bemounted on second retail shelving unit 604. Consistent with the presentdisclosure, image capture device 506 may be configured relative to firsthousing 502 such that an optical axis of image capture device 506 isdirected toward an opposing retail shelving unit when first housing 502is fixedly mounted on a retail shelving unit. For example, optical axis606 of the image capture device associated with first housing 502B maybe directed towards second retail shelving unit 604 when first housing502E3 is fixedly mounted on first retail shelving unit 602. A singleretail shelving unit may hold a number of systems 500 that include aplurality of image capturing devices. Each of the image capturingdevices may be associated with a different field of view directed towardthe opposing retail shelving unit. Different vantage points ofdifferently located image capture devices may enable image acquisitionrelative to different sections of a retail shelf. For example. at leastsome of the plurality of image capturing devices may be fixedly mountedon shelves at different heights. Examples of such a deployment areillustrated in FIGS. 4A and 6B.

As shown in FIG. 6A each first housing 502 may be associated with a dataconduit 508 that enables exchanging of information (e.g., image data,control signals, etc.) between the at least one processor located insecond housing 504 and image capture device 506 located in first housing502. In some embodiments, data conduit 508 may include a wiredconnection that supports data-transfer and may he used to power imagecapture device 506 (e.g., data conduit 508A, data conduit 508B, dataconduit 508D1, data conduit 508D2, data conduit 508E1, and data conduit508E2). Consistent with these embodiments, data conduit 508 may complywith a wired standard such as USB, Micro-USB, HDMI, Micro-HDMI,Firewire, Apple, etc. In other embodiments, data conduit 508 may be awireless connection, such as a dedicated communications channel betweenthe at least one processor located in second housing 504 and imagecapture device 506 located in first housing 502 (e.g., data conduit508C). In one example, the communications channel may be established bytwo Near Field Communication (NFC) transceivers. In other examples,first housing 502 and second housing 504 may include interface circuitsthat comply with other short-range wireless standards such as Bluetooth,WiFi, ZigBee, etc.

In some embodiments of the disclosure, the at least one processor ofsystem 500 may cause at least one image capture device 506 toperiodically capture images of products located on an opposing retailshelving unit (e.g., images of products located on a shelf across anaisle from the shelf on which first housing 502 is mounted). The term“periodically capturing images” includes capturing art image or imagesat predetermined time intervals (e.g., every minute, every 30 minutes,every 150 minutes, every 300 minutes, etc.), capturing video, capturingan image every time a status request is received, and/or capturing animage subsequent to receiving input from an additional sensor, forexample, an associated proximity sensor. Images may also be capturedbased on various other triggers or in response to various other detectedevents. In some embodiments, system 500 may receive an output signalfrom at least one sensor located on an opposing retail shelving unit.For example, system 500B may receive output signals from a sensingsystem located on second retail shelving unit 604. The output signalsmay be indicative of a sensed lining of a product from second retailshelving unit 604 or a sensed positioning of a product on second retailshelving unit 604. In response to receiving the output signal from theat least one sensor located on second retail shelving unit 604, system500B may cause image capture device 506 to capture one or more images ofsecond retail shelving unit 604. Additional details on a sensing system,including the at least one sensor that generates output signalsindicative of a sensed lifting of a product from an opposing retailshelving unit, is discussed below with reference to FIGS. 8-10.

Consistent with embodiments of the disclosure, system 500 may detect anobject 608 in a selected area between first retail shelving unit 602 andsecond retail shelving unit 604. Such detection may be based on theoutput of one or more dedicated sensors (e.g., motion detectors, etc.)and/or may be based on image analysis of one or more images acquired byan image acquisition device. Such images, for example, may include arepresentation of a person or other object recognizable through variousimage analysis techniques (e.g., trained neural networks, Fouriertransform analysis, edge detection, filters, face recognition, etc.).The selected area may be associated with distance di between firstretail shelving unit 602 and second retail shelving unit 604. Theselected area may be within the field of view of image capture device506 or an area where the object causes an occlusion of a region ofinterest (such as a shelf, a portion of a shelf being monitored, andmore). Upon detecting object 608, system 500 may cause image capturedevice 506 to forgo image acquisition while object 608 is within theselected area. In one example, object 608 may be an individual, such asa customer or a store employee. In another example, detected object 608may be an inanimate object, such as a cart, box, carton, one or moreproducts, cleaning robots, etc. In the example illustrated in FIG. 6A,system 500A may detect that object 608 has entered into its associatedfield of view (e.g,, using a proximity sensor) and may instruct imagecapturing device 506 to forgo image acquisition. In alternativeembodiments, system 500 may analyze a plurality of images acquired byimage capture device 506 and identify at least one image of theplurality of images that includes a representation of object 608.Thereafter, system 500 may avoid transmission of at least part of the atleast one identified image and/or information based on the at least oneidentified image to server 135.

As shown in FIG. 6A, the at least one processor contained in a secondhousing 504 may control a plurality of image capture devices 506contained in a plurality of first housings 502 (e.g., systems 500D and500E). Controlling image capturing device 506 may include instructingimage capturing device 506 to capture an image and/or transmit capturedimages to a remote server (e.g., server 135). In sonic cases, each ofthe plurality of image capture devices 506 may have a field of view thatat least partially overlaps with a field of view of at least one otherimage capture device 506 from among plurality of image capture devices506. In one embodiment, the plurality of image capture devices 506 maybe configured for location on one or more horizontal shelves and may bedirected to substantially different areas of the opposing first retailshelving unit. In this embodiment, the at least one processor maycontrol the plurality of image capture devices such that each of theplurality of image capture devices may capture an image at a differenttime. For example, system 500E may have a second housing 504E with atleast one processor that may instruct a first image capturing devicecontained in first housing 502E1 to capture an image at a first time andmay instruct a second image capturing device contained in first housing502E2 to capture an image at a second time which differs from the firsttime. Capturing images in different times (or forwarding them to the atleast one processor at different times) may assist in processing theimages and writing the images in the memory associated with the at leastone processor.

FIG. 6B illustrates a perspective view assembly diagram depicting aportion of a retail shelving unit 620 with multiple systems 500 (e,g.,500F, 500G, 500H, 500I, and 500J) deployed thereon for acquiring imagesof products. Retail shelving unit 620 may include horizontal shelves atdifferent heights. For example, horizontal shelves 622A, 62213, and 622Care located below horizontal shelves 622D, 622E, and 622F. In someembodiments, a different number of systems 500 may be mounted on shelvesat different heights. For example, system 500F (including first housing502F and second housing 504F), system 500G (including first housing 502Gand second housing 504G), and system 500H (including first housing 502Hand second housing 5041-1) may be mounted on horizontal shelvesassociated with a first height; and system 5001 (including first housing5021, second housing 504J, and a projector 632) and system 500J(including first housing 502J1, first housing 502J2, and second housing504J) may be mounted on horizontal shelves associated with a secondheight. In some embodiments, retail shelving unit 620 may include ahorizontal shelf with at least one designated place (not shown) formounting a housing of image capturing device 506. The at least onedesignated place may be associated with connectors such that firsthousing 502 may be fixedly mounted on a side of horizontal shelf 622facing an opposing retail shelving unit using the connectors.

Consistent with the present disclosure, system 500 may be mounted on aretail shelving unit that includes at least two adjacent horizontalshelves (e.g., shelves 622A and 622B) forming a substantially continuoussurface for product placement. The store shelves may include standardstore shelves or customized store shelves. A length of each store shelf622 may be at least 50 cm, less than 200 cm, or between 75 cm to 175 cm.In one embodiment, first housing 502 may be fixedly mounted on theretail shelving unit in a slit between two adjacent horizontal shelves.For example, first housing 502G may be fixedly mounted on retailshelving unit 620 in a slit between horizontal shelf 622B and horizontalshelf 622C. In another embodiment, first housing 502 may be fixedlymounted on a first shelf and second housing 504 may be fixedly mountedon a second shelf. For example, first housing 502I may be mounted onhorizontal shelf 622D and second housing 504I may be mounted onhorizontal shelf 622E. In another embodiment, first housing 502 may befixedly mounted on a retail shelving unit on a first side of ahorizontal shelf facing the opposing retail shelving unit and secondhousing 504 may be fixedly mounted on retail shelving unit 620 on asecond side of the horizontal shelf orthogonal to the first side. Forexample, first housing 502H may mounted on a first side 624 ofhorizontal shelf 622C next to a label and second housing 5041I may bemounted on a second side 626 of horizontal shelf 622C that faces down(e.g., towards the ground or towards a lower shelf). In anotherembodiment, second housing 504 may be mounted closer to the back of thehorizontal shelf than to the front of the horizontal shelf. For example,second housing 504H may be fixedly mounted on horizontal shelf 622C onsecond side 626 closer to third side 628 of the horizontal shelf 622Cthan to first side 624. Third side 628 may be parallel to first side624. As mentioned above, data conduit 508 (e.g., data conduit 508H) mayhave an adjustable or selectable length for extending between firsthousing 502 and second housing 504. In one embodiment, when firsthousing 502H is fixedly mounted on first side 624, the length of dataconduit 508H may enable second housing 604H to be fixedly mounted onsecond side 626 closer to third side 628 than to first side 624.

As mentioned above, at least one processor contained in a single secondhousing 504 may control a plurality of image capture devices 506contained in a plurality of first housings 502 (e.g., system 5004 Insome embodiments, the plurality of image capture devices 506 may beconfigured for location on a single horizontal shelf and may be directedto substantially the same area of the opposing first retail shelvingunit (e.g., system 500D in FIG. 6A). In these embodiments, the imagedata acquired by the first image capture device and the second imagecapture device may enable a calculation of depth information (e.g.,based on image parallax information) associated with at least oneproduct positioned on an opposing retail shelving unit. For example,system 500J may have single second housing 504J with at least oneprocessor that may control a first image capturing device contained infirst housing 502J1 and a second image capturing device contained infirst housing 502J2. The: distance d3 between the first image capturedevice contained in first housing 502J1 and the second image capturedevice contained in first housing 502J2 may be selected based on thedistance between retail shelving unit 620 and the opposing retailshelving unit (e.g., similar to d1 and/or d2). For example, distance d3may be at least 5 cm, at least 10 cm, at least 15 cm, less than 40 cm,less than 30 cm, between about 5 cm to about 20 cm, or between about 10cm to about 15 cm. In another example, d3 may be a function of d1 and/ord2, a linear function of d1 and/or d2, a Function of d1*log(d1) and/ord2*log(d2) such as a1*d1*log(d1) for some constant a1, and so forth. Thedata from the first image capturing device contained in first housing502J1 and the second image capturing device contained in first housing502J2 may be used to estimate the number of products on a store shelf ofretail shelving unit 602. In related embodiments, system 500 may controla projector (e.g., projector 632) and image capture device 506 that areconfigured tor location on a single store shelf or on two separate storeshelves. For example, projector 632 may be mounted on horizontal shelf622E and image capture device 506I may be mounted on horizontal shelf622D. The image data acquired by image capture device 506 (e.g.,included in first housing 502I) may include reflections of lightpatterns projected from projector 632 on the at least one product and/orthe opposing retail shelving unit and may enable a calculation of depthinformation associated with at least one product positioned on theopposing retail shelving unit. The distance between projector 632 andthe image capture device contained in first housing 5021 may be selectedbased on the distance between retail shelving unit 620 and the opposingretail shelving unit (e.g., similar to d1 and/or d2). For example, thedistance between the projector and the image capture device may be atleast 5 cm, at least 10 cm, at least 15 cm, less than 40 cm, less than30 cm, between about 5 cm to about 20 cm, or between about 10 cm toabout 15 cm. In another example, the distance between the projector andthe image capture device may be a function of d1 and/or d2, a linearfunction of d1 and/or d2, a function of d1*log(d1) and/or d2*log(d2)such as a2*d1*log(d1) for some constant a1, and so forth.

Consistent with the present disclosure, a central communication device630 may he located in retail store 105 and may be configured tocommunicate with server 135 (e.g., via an Internet connection). Thecentral communication device may also communicate with a plurality ofsystems 500 (for example, less than ten, ten, eleven, twelve, more thantwelve, and so forth). In some cases, at least one system of theplurality of systems 500 may be located in proximity to centralcommunication device 630. In the illustrated example, system 500F may belocated in proximity to central communication device 630. In someembodiments, at least some of systems 500 may communicate directly withat least one other system 500. The communications between some of theplurality of systems 500 may happen via a wired connection, such as thecommunications between system 500J and system 500I and thecommunications between system 500H and system 500G. Additionally oralternatively, the communications between some of the plurality ofsystems 500 may occur via a wireless connection, such as thecommunications between system 500G and system 500F and thecommunications between system 500I and system 500F. In some examples, atleast one system 500 may be configured to transmit captured image data(or information derived from the captured image data) to centralcommunication device 630 via at least two mediating systems 500, atleast three mediating systems 500, at least four mediating systems 500,or more. For example, system 500J may convey captured image data tocentral communication device 630 via system 500I and system 500F.

Consistent with the present disclosure, two (or more) systems 500 mayshare information to improve image acquisition. For example, system 500Jmay be configured to receive from a neighboring system 500I informationassociated with an event that system 500I had identified, and controlimage capture device 506 based on the received information. For example,system 500I may forgo image acquisition based on an indication fromsystem 500I that an object has entered or is about to enter its field ofview. Systems 500I and 500J may have overlapping fields of view ornon-overlapping fields of view. In addition, system 500J may alsoreceive (from system 500J) information that originates from centralcommunication device 630 and control image capture device 506 based onthe received information. For example, system 500I may receiveinstructions from central communication device 630 to capture an imagewhen suppler 115 inquiries about a specific product that is placed in aretail unit opposing system 500I. In some embodiments, a plurality ofsystems 500 may communicate with central communication device 630. Inorder to reduce or avoid network congestion, each system 500 mayidentify an available transmission time slot. Thereafter, each system500 may determine a default time slot for future transmissions based onthe identified transmission time slot.

FIG. 6C provides a diagrammatic representation of a retail shelving unit640 being captured by multiple systems 500 (e.g., system 500K and system500L) deployed on an opposing retail shelving unit (not shown). FIG. 6Cillustrates embodiments associated with the process of installingsystems 500 in retail store 105. To facilitate the installation ofsystem 500, each first housing 502 (e.g., first housing 502K) mayinclude an adjustment mechanism 642 for setting a field of view 644 ofimage capture device 506K such that the field of view 644 will at leastpartially encompass products placed both on a bottom shelf of retailshelving unit 640 and on a top shelf of retail shelving unit 640,. Forexample, adjustment mechanism 642 may enable setting the position ofimage capture device 506K relative to first housing 502K. Adjustmentmechanism 642 may have at least two degrees of freedom to separatelyadjust manually (or automatically) the vertical field of view and thehorizontal field of view of image capture device 506K. In oneembodiment, the angle of image capture device 506K may be measured usingposition sensors associated with adjustment mechanism 642, and themeasured orientation may be used to determine if image capture device506K is positioned in the right direction. In one example, the output ofthe position sensors may be displayed on a handheld device of anemployee installing image capturing device 506K. Such an arrangement mayprovide the employee/installer with real time visual feedbackrepresentative of the field of view of an image acquisition device beinginstalled.

In addition to adjustment mechanism 642, first housing 502 may include afirst physical adapter (not shown) configured to operate with multipletypes of image capture device 506 and a second physical adapter (notshown) configured to operate with multiple types of lenses. Duringinstallation, the first physical adapter may be used to connect asuitable image capture device 506 to system 500 according to the levelof recognition requested detecting a barcode from products, detectingtext and price from labels, detecting different categories of products,etc.). Similarly, during installation, the second physical adapter maybe used to associate a suitable lens to image capture device 506according to the physical conditions at the store (e.g,, the distancebetween the aisles, the horizontal field of view required from imagecapture device 506, and/or the vertical field of view required fromimage capture device 506). The second physical adapter provides theemployee/installer the ability to select the focal length of lens 312during installation according to the distance between retail shelvingunits on opposite sides of an aisle (e.g., distance d1 and/or distanced2 shown in FIG. 6A). In some embodiments, adjustment mechanism 642 mayinclude a locking mechanism to reduce the likelihood of unintentionalchanges in the field of view of image capture device 506. Additionallyor alternatively, the at least one processor contained in second housing504 may detect changes in the field of view of image capture device 506and issue a warning when a change is detected, when a change larger thana selected threshold is detected, when a change is detected for aduration longer than a selected threshold, and so forth.

In addition to adjustment mechanism 642 and the different physicaladapters, system 500 may modify the image data acquired by image capturedevice 506 based on at least one attribute associated with opposingretail shelving unit 640. Consistent with the present disclosure, the atleast one attribute associated with retail shelving unit 640 may includea lighting condition, the dimensions of opposing retail shelving unit640, the size of products displayed on opposing retail shelving unit640, the type of labels used on opposing retail shelving unit 640, andmore. In some embodiments, the attribute may be determined, based onanalysis of one or more acquired images, by at least one processorcontained in second housing 504. Alternatively, the attribute may beautomatically sensed and conveyed to the at least one processorcontained in second housing 504. In one example, the at least oneprocessor may change the brightness of captured images based on thedetected light conditions. In another example, the at least oneprocessor may modify the image data by cropping the image such that itwill include only the products on retail shelving unit (e.g., not toinclude the floor or the ceiling), only area of the shelving unitrelevant to a selected task (such as planogram compliance check), and soforth.

Consistent with the present disclosure, during installation, system 500may enable real-time display 646 of field of view 644 on a handhelddevice 648 of a user 650 installing image capturing device 506K. In oneembodiment, real-time display 646 of field of view 644 may includeaugmented markings 652 indicating a location of a field of view 654 ofan adjacent image capture device 506L. In another embodiment, real-timedisplay 646 of field of view 644 may include augmented markings 656indicating a region of interest in opposing retail shelving unit 640.The region of interest may be determined based on a planogram,identified product type, and/or part of retail shelving unit 640. Forexample, the region of interest may include products with a greaterlikelihood of planogram incompliance. In addition, system 500K mayanalyze acquired images to determine if field of view 644 includes thearea that image capturing device 506K is supposed to monitor (forexample, from labels on opposing retail shelving unit 640, products onopposing retail shelving unit 640, images captured from other imagecapturing devices that may capture other pads of opposing retailshelving unit 640 or capture the same part of opposing retail shelvingunit 640 but in a lower resolution or at a lower frequency, and soforth). In additional embodiments, system 500 may further comprise anindoor location sensor which may help determine if the system 500 ispositioned at the right location in retail store 105.

In some embodiments, an anti-theft device may be located in at least oneof first housing 502 and second housing 504. For example, the anti-theftdevice may include a specific RF label or a pin-tag radio-frequencyidentification device, which may be the same or similar to a type ofanti-theft device that is used by retail store 105 in which system 500is located. The RF label or the pin-tag may be incorporated within thebody of first housing 502 and second housing 504 and may not be visible.In another example, the anti-theft device may include a motion sensorwhose output may be used to trigger an alarm in the case of motion ordisturbance, in case of motion that is above a selected threshold, andso forth.

FIG. 7A includes a flowchart representing an exemplary method 700 foracquiring images of products in retail store 105 in accordance withexample embodiments of the present disclosure, For purposes ofillustration, in the following description, reference is made to certaincomponents of system 500 as deployed in the configuration depicted inFIG. 6A. It will be appreciated, however, that other implementations arepossible and that other configurations may be utilized to implement theexemplary method. It will also be readily appreciated that theillustrated method can be altered to modify the order of steps, deletesteps, or further include additional steps.

At step 702, the method includes fixedly mounting on first retailshelving unit 602 at least one first housing 502 containing at least oneimage capture device 506 such that an optical axis (e.g., optical axis606) of at least one image capture device 506 is directed to secondretail shelving unit 604. In one embodiment, fixedly mounting firsthousing 502 on first retail shelving unit 602 may include placing firsthousing 502 on a side of store shelf 622 facing second retail shelvingunit 604. In another embodiment, fixedly mounting first housing 502 onretail shelving unit 602 may include placing first housing 502 in a slitbetween two adjacent horizontal shelves. In some embodiments, the methodmay further include fixedly mounting on first retail shelving unit 602at least one projector (such as projector 632) such that light patternsprojected by the at least one projector are directed to second retailshelving unit 604. In one embodiment, the method may include mountingthe at least one projector to first retail shelving unit 602 at aselected distance to first housing 502 with image capture device 506. Inone embodiment, the selected distance may be at least 5 cm, at least 10cm, at least 15 cm, less than 40 cm, less than 30 cm, between about 5 cmto about 20 cm, or between about 10 cm to about 15 cm. In oneembodiment, the selected distance may be calculated according to adistance between to first retail shelving unit 602 and second retailshelving unit 604, such as d1 and/or d2, for example selecting thedistance to be a function of d1 and/or d2, a linear function of d1and/or d2, a function of d1*log(d1) and/or d2*log(d2) such asa1*d1*log(d1) for some constant a1, and so forth.

At step 704, the method includes fixedly mounting on first retailshelving unit 602 second housing 504 at a location spaced apart from theat least one first housing 502, second housing 504 may include at leastone processor (e.g., processing device 302). In one embodiment, fixedlymounting second housing 504 on the retail shelving unit may includeplacing second housing 504 on a different side of store shelf 622 thanthe side first housing 502 is mounted on.

At step 706, the method includes extending at least one data conduit 508between at least one first housing 502 and second housing 504. In oneembodiment, extending at least one data conduit 508 between at least onefirst housing 502 and second housing 504 may include adjusting thelength of data conduit 508 to enable first housing 502 to be mountedseparately from second housing 504. At step 708, the method includescapturing images of second retail shelving unit 604 using at least oneimage capture device 506 contained in at least one first housing 502(e.g., first housing 502A, first housing 502B, or first housing 502C).In one embodiment, the method further includes periodically capturingimages of products located on second retail shelving unit 604. Inanother embodiment the method includes capturing images of second retailshelving unit 604 after receiving a trigger from at least one additionalsensor in communication with system 500 (wireless or wired).

At step 710, the method includes transmitting at least some of thecaptured images from second housing 504 to a remote server (e.g., server135) configured to determine planogram compliance relative to secondretail shelving unit 604. In some embodiments, determining planogramcompliance relative to second retail shelving unit 604 may includedetermining at least one characteristic of planogram compliance based ondetected differences between the at least one planogram and the actualplacement of the plurality of product types on second retail shelvingunit 604. Consistent with the present disclosure, the characteristic ofplanogram compliance may include at least one of: product facing,product placement, planogram compatibility, price correlation, promotionexecution, product homogeneity, restocking rate, and planogramcompliance of adjacent products.

FIG. 7B provides a flowchart representing an exemplary method 720 foracquiring images of products in retail store 105, in accordance withexample embodiments of the present disclosure. For purposes ofillustration, in the following description, reference is made to certaincomponents of system 500 as deployed in the configuration depicted inFIG. 6A. It will be appreciated, however, that other implementations arepossible and that other configurations may be utilized to implement theexemplary method. It will also be readily appreciated that theillustrated method can be altered to modify the order of steps, deletesteps, or further include additional steps.

At step 722, at least one processor contained in a second housing mayreceive from at least one image capture device contained in at least onefirst housing fixedly mounted on a retail shelving unit a plurality ofimages of an opposing retail shelving unit. For example, at least oneprocessor contained in second housing 504A may receive from at least oneimage capture device 506 contained in first housing 502A (fixedlymounted on first retail shelving unit 602) a plurality of images ofsecond retail shelving unit 604. The plurality of images may be capturedand collected during a period or time (e.g., a minute, an hour, sixhours, a day, a week, or more).

At step 724, the at least one processor contained in the second housingmay analyze the plurality of images acquired by the at least one imagecapture device. In one embodiment, at least one processor contained insecond housing 504A may use any suitable image analysis technique (forexample, object recognition, object detection, image sementation,feature extraction, optical character recognition (OCR), object-basedimage analysis, shape region techniques, edge detection techniques,pixel-based detection, artificial neural networks, convolutional neuralnetworks, etc.) to identify objects in the plurality of images. In oneexample, the at least one processor contained in second housing 504A maydetermine the number of products located in second retail shelving unit604. In another example, the at least one processor contained in secondhousing 504A may detect one or more objects in an area between firstretail shelving unit 602 and second retail shelving unit 604.

At step 726, the at least one processor contained in the second housingmay identify in the plurality of images a first image that includes arepresentation of at least a portion of an object located in an areabetween the retail shelving unit and the opposing retail shelving unit.In step 728, the at least one processor contained in the second housingmay identify in the plurality of images a second image that does notinclude any object located in an area between the retail shelving unitand the opposing retail shelving unit. In one example, the object in thefirst image may be an individual, such as a customer or a storeemployee. In another example, the object in the first image may be aninanimate object, such as carts, boxes, products, etc.

At step 730, the at least one processor contained in the second housingmay instruct a network interface contained in the second housing,fixedly mounted on the retail shelving unit separate from the at leastone first housing, to transmit the second image to a remote server andto avoid transmission of the first image to the remote server. Inaddition, the at least one processor may issue a notification when anobject blocks the field of view of the image capturing device for morethan a predefined period of time (e.g., at least 30 minutes, at least 75minutes, at least 150 minutes).

Embodiments of the present disclosure may automatically assesscompliance of one or more store shelves with a planogram. For example,embodiments of the present disclosure may use signals from one or moresensors to determine placement of one or more products on store shelves.The disclosed embodiments may also use one or more sensors to determineempty spaces on the store shelves. The placements and empty spaces maybe automatically assessed against a digitally encoded planogram. Aplanogram refers to any data structure or specification that defines atleast one product characteristic relative to a display structureassociated with a retail environment (such as store shelf or area of oneor more shelves). Such product characteristics may include, among otherthings, quantities of products with respect to areas of the shelves,product configurations or product shapes with respect to areas of theshelves, product arrangements with respect to areas of the shelves,product density with respect to areas of the shelves, productcombinations with respect to areas of the shelves, etc. Althoughdescribed with reference to store shelves, embodiments of the presentdisclosure may also be applied to end caps or other displays; bins,shelves, or other organizers associated with a refrigerator or freezerunits; or any other display structure associated with a retailenvironment.

The embodiments disclosed herein may use any sensors configured todetect one or more parameters associated with products (or a lackthereof). For example, embodiments may use one or more of pressuresensors, weight sensors, light sensors, resistive sensors, capacitivesensors, inductive sensors, vacuum pressure sensors, high pressuresensors, conductive pressure sensors, infrared sensors, photo-resistorsensors, photo-transistor sensors, photo-diodes sensors, ultrasonicsensors, or the like. Some embodiments may use a plurality of differentkinds of sensors, for example, associated with the same or overlappingareas of the shelves and/or associated with different areas of theshelves. Some embodiments may use a plurality of sensors configured tobe placed adjacent a store shelf, configured for location on the storeshelf, configured to be attached to, or configured to be integrated withthe store shelf, in some cases, at least part of the plurality ofsensors may be configured to be placed next to a surface of a storeshelf configured to hold products. For example, the at least part of theplurality of sensors may be configured to be placed relative to a partof a store shelf such that the at least part of the plurality of sensorsmay be positioned between the part of a store shelf and products placedon the part of the shelf. In another embodiment, the at least part ofthe plurality of sensors may be configured to be placed above and/orwithin and/or under the part of the shelf.

In one example, the plurality of sensors may include light detectorsconfigured to be located such that a product placed on the pan of theshelf may block at least some of the ambient light from reaching thelight detectors. The data received from the light detectors may beanalyzed to detect a product or to identify a product based on the shapeof a product placed on the part of the shelf. In one example, the systemmay identify the product placed above the light detectors based on datareceived from the light detectors that may be indicative of at leastpart of the ambient light being blocked from reaching the lightdetectors. Further, the data received from the light detectors may beanalyzed to detect vacant spaces on the store shelf. For example, thesystem may detect vacant spaces on the store shelf based on the receiveddata that may be indicative of no product being placed on a part of theshelf. In another example, the plurality of sensors may include pressuresensors configured to be located such that a product placed on the partof the shelf may apply detectable pressure on the pressure sensors.Further, the data received from the pressure sensors may be analyzed todetect a product or to identify a product based on the shape of aproduct placed on the part of the shelf. In one example, the system mayidentify the product placed above the pressure sensors based on datareceived from the pressure sensors being indicative of pressure beingapplied on the pressure sensors. In addition, the data from the pressuresensors may be analyzed to detect vacant spaces on the store shelf, forexample based on the readings being indicative of no product beingplaced on a part of the shelf, for example, when the pressure readingsare below a selected threshold. Consistent with the present disclosure,inputs from different types of sensors (such as pressure sensors, lightdetectors, etc.) may be combined and analyzed together, for example todetect products placed on a store shelf, to identify shapes of productsplaced on a store shelf, to identify types of products placed on a storeshelf, to identify vacant spaces on a store shelf, and so forth.

With reference to FIG. 8A and consistent with the present disclosure, astore shelf 800 may include a plurality of detection elements, e.g.,detection elements 801A and 801B. In the example of FIG. 8A, detectionelements 801A and 801B may comprise pressure sensors and/or other typeof sensors for measuring one or more parameters (such as resistance,capacitance, or the like) based on physical contact (or lack thereof)with products, e.g., product 803A and product 803B. Additionally oralternatively, detection elements configured to measure one or moreparameters (such as current induction, magnetic induction, visual orother electromagnetic reflectance, visual or other electromagneticemittance, or the like) may be included to detect products based onphysical proximity (or lack thereof) to products. Consistent with thepresent disclosure, the plurality of detection elements may beconfigured for location on shelf 800. The plurality of detectionelements may be configured to detect placement of products when theproducts are placed above at least part of the plurality of detectionelements. Some embodiments of the disclosure, however, may be performedwhen at least some of the detection elements may be located next toshelf 800 (e.g., for magnetometers or the like), across from shelf 800(e.g., for image sensors or other light sensors, light detection andranging (LIDAR) sensors, radio detection and ranging (RADAR) sensors, orthe like), above shelf 800 (e.g., for acoustic sensors or the like),below shelf 800 (e.g., for pressure sensors or the like), or any otherappropriate spatial arrangement. Although depicted as standalone unitsin the example of FIG. 8A, the plurality of detection elements may formpart of a fabric (e.g., a smart fabric or the like), and the fabric maybe positioned on a shelf to take measurements. For example, two or moredetection elements may be integrated together into a single structure(e.g., disposed within a common housing, integrated together within afabric or mat, etc.). In some examples, detection elements (such asdetection elements 801A and 801B may be placed adjacent to (or placedon) store shelves as described above. Some examples of detectionelements may include pressure sensors and/or light detectors configuredto be placed above and/or within and/or under a store shelf as describedabove.

Detection elements associated with shelf 800 may be associated withdifferent areas of shelf 800. For example, detection elements 801A and801B are associated with area 805A while other detection elements areassociated with area 805B. Although depicted as rows, areas 805A and805B may comprise any areas of shelf 800, whether contiguous (e.g., asquare, a rectangular, or other regular or irregular shape) or not(e.g., a plurality of rectangles or other regular and/or irregularshapes). Such areas may also include horizontal regions between shelves(as shown in FIG. 8A) or may include vertical regions that include areaof multiple different shelves (e.g., columnar regions spanning overseveral different horizontally arranged shelves). In some examples, theareas may be part of a single plane. In some examples, each area may bepart of a different plane. In some examples, a single area may be partof a single plane or be divided across multiple planes.

One or more processors (e.g., processing device 202) configured tocommunicate with the detection elements (e.g., detection elements 801Aand 801B may detect first signals associated with a first area (e.g.,areas 805A and/or 805B) and second signals associated with a secondarea. In some embodiments, the first area may, in part, overlap with thesecond area. For example, one or more detection elements may beassociated with the first area as well as the second area and/or one ormore detection elements of a first type may be associated with the firstarea while one or more detection elements of a second type may beassociated with the second area overlapping, at least in part, the firstarea. In other embodiments, the first area and the second area may bespatially separate from each other.

The one or more processors may, using the first and second signals,determine that one or more products have been placed in the first areawhile the second area includes at least one empty area. For example, ifthe detection elements include pressure sensors, the first signals mayinclude weight signals that match profiles of particular products (suchas the mugs or plates depicted in the example of FIG. 8A), and thesecond signals may include weight signals indicative of the absence ofproducts (e.g., by being equal to or within a threshold of a defaultvalue such as atmospheric pressure or the like). The disclosed weightsignals may be representative of actual weight values associated with aparticular product type or, alternatively, may be associated with arelative weight value sufficient to identify the product and/or toidentify the presence of a product. In some cases, the weight signal maybe suitable for verifying the presence of a product regardless ofwhether the signal is also sufficient for product identification. Inanother example, if the detection elements include light detectors (asdescribed above), the first signals may include light signals that matchprofiles of particular products (such as the mugs or plates depicted inthe example of FIG. 8A), and the second signals may include lightsignals indicative of the absence of products (e.g., by being equal toor within a threshold of a default value such as values corresponding toambient light or the like). For example, the first light signals may beindicative of ambient light being blocked by particular products, whilethe second light signals may be indicative of no product blocking theambient light The disclosed light signals may be representative ofactual light patterns associated with a particular product type or,alternatively, may be associated with light patterns sufficient toidentify the product and/or to identify the presence of a product.

The one or more processors may similarly process signals from othertypes of sensors. For example, if the detection elements includeresistive or inductive sensors, the first signals may includeresistances, voltages, and/or currents that match profiles of particularproducts (such as the mugs or plates depicted in the example of FIG. 8Aor elements associated with the products, such as tags, etc.), and thesecond signals may include resistances, voltages, and/or currentsindicative of the absence of products (e.g., by being equal to or withina threshold of a default value such as atmospheric resistance, a defaultvoltage, a default current, corresponding to ambient light, or thelike). In another example, if the detection elements include acoustics,LIDAR, RADAR, or other reflective sensors, the first signals may includepatterns of returning waves (whether sound, visible light, infraredlight, radio, or the like) that match profiles of particular products(such as the mugs or plates depicted in the example of FIG. 8A), and thesecond signals may include patterns of returning waves (whether sound,visible light, infrared light, radio, or the like) indicative of theabsence of products (e.g., by being equal to or within a threshold of apattern associated with an empty shelf or the like).

Any of the profile matching described above may include direct matchingof a subject to a threshold. For example, direct matching may includetesting one or more measured values against the profile value(s) withina margin of error; mapping a received pattern onto a profile patternwith a residual having a maximum, minimum, integral, or the like withinthe margin of error; performing an autocorrelation, Fourier transform,convolution, or other operation on received measurements or a receivedpattern and comparing the resultant values or function against theprofile within a margin of error; or the like. Additionally oralternatively, profile matching may include fuzzy matching betweenmeasured values and/or patterns and a database of profiles such that aprofile with a highest level of confidence according to the fuzzysearch. Moreover, as depicted in the example of FIG. 8A, products, suchas product 803B, may be stacked and thus associated with a differentprofile when stacked than when standalone.

Any of the profile matching described above may include use of one ormore machine learning techniques. For example, one or more artificialneural networks, random forest models, or other models trained onmeasurements annotated with product identifiers may process themeasurements from the detection elements and identify productstherefrom. In such embodiments, the one or more models may useadditional or alternative input, such as images of the shelf (e.g., fromcapturing devices 125 of FIGS. 4A-4C explained above) or the like.

Based on detected products and/or empty spaces, determined using thefirst signals and second signals, the one or more processors maydetermine one or more aspects of planogram compliance. For example, theone or more processors may identify products and their locations on theshelves, determine quantities of products within particular areas (e.g.,identifying stacked or clustered products), identity facing directionsassociated with the products (e.g., whether a product is outward facing,inward lacing, askew, or the like), or the like. Identification of theproducts may include identifying a product type (e.g,, a bottle of soda,a loaf of broad, a notepad, or the like) and/or a product brand (e.g., aCoca-Cola® bottle instead of a Sprite® bottle, a Starbucks® coffeetumbler instead of a Tervise coffee tumbler, or the like). Productfacing direction and/or orientation, for example, may be determinedbased on a detected orientation of an asymmetric shape of a product baseusing pressure sensitive pads, detected density of products, etc. Forexample, the product facing may be determined based on locations ofdetected product bases relative to certain areas of a shelf (e.g., alonga front edge of a shelf), etc. Product facing may also be determinedusing image sensors, light sensors, or any other sensor suitable fordetecting product orientation.

The one or more processors may generate one or more indicators of theone or more aspects of planogram compliance. For example, an indicatormay comprise a data packet, a data file, or any other data structureindicating any variations from a planogram, e.g., with respect toproduct placement such as encoding intended coordinates of a product andactual coordinates on the shelf, with respect to product facingdirection and/or orientation such as encoding indicators of locationsthat have products not facing a correct direction and/or in an undesiredorientation, or the like.

In addition to or as an alternative to determining planogram compliance,the one or more processors may detect a change in measurements from oneor more detection elements. Such measurement changes may trigger aresponse. For example, a change of a first type may trigger capture ofat least one image of the shelf (e.g., using capturing devices 125 ofFIGS. 4A-4C explained above) while a detected change of a second typemay cause the at least one processor to forgo such capture. A first typeof change may, for example, indicate the moving of a product from onelocation on the shelf to another location such that planogram compliancemay be implicated. In such cases, it may be desired to capture an imageof the product rearrangement in order to assess or reassess productplanogram compliance. In another example, a first type of change mayindicate the removal of a product from the shelf, e.g., by an employeedue to damage, by a customer to purchase, or the like. On the otherhand, a second type of change may, for example, indicate the removal andreplacement of a product to the same (within a margin of error) locationon the shelf, e.g., by a customer to inspect the item. In cases whereproducts are removed from a shelf, but then replaced on the shelf (e.g.,within a particular time window), the system may forgo a new imagecapture, especially if the replaced product is detected in a locationsimilar to or the same as its recent, original position.

With reference to FIG. 8B and consistent with the present disclosure, astore shelf 850 may include a plurality of detection elements, e.g.,detection elements 851A and 851B. In the example of FIG. 8B, detectionelements 851A and 851B may comprise light sensors and/or other sensorsmeasuring one or more parameters (such as visual or otherelectromagnetic reflectance, visual or other electromagnetic emittance,or the like) based on electromagnetic waves from products, product 853Aand product 853B. Additionally or alternatively, as explained above withrespect to FIG. 8B, detection elements 851A and 851B may comprisepressure sensors, other sensors measuring one or more parameters (suchas resistance, capacitance, or the like) based on physical contact (orlack thereof) with the products, and/or other sensors that measure oneor more parameters (such as current induction, magnetic induction,visual or other electromagnetic reflectance, visual or otherelectromagnetic emittance, or the like) based on physical proximity (orlack thereof) to products.

Moreover, although depicted as located on shelf 850, some detectionelements may be located next to shelf 850 (e.g., for magnetometers orthe like), across from shelf 850 (e.g., for image sensors or other lightsensors, light detection and ranging (LIDAR) sensors, radio detectionand ranging (RADAR) sensors, or the like), above shelf 850 (e.g., foracoustic sensors or the like), below shelf 850 (e.g., for pressuresensors, light detectors, or the like), or any other appropriate spatialarrangement. Further, although depicted as standalone in the example ofFIG. 8B, the plurality of detection elements may form part of a fabric(e.g., a smart fabric or the like), and the fabric may be positioned ona shelf to take measurements.

Detection elements associated with shelf 850 may be associated withdifferent areas of shelf 850, e.g., area 855A, area 855B, or the like.Although depicted as rows, areas 855A and 855B may comprise any areas ofshelf 850, whether contiguous (e.g., a square, a rectangular, or otherregular or irregular shape) or not (e.g., a plurality of rectangles orother regular and/or irregular shapes).

One or more processors (e.g., processing device 202) in communicationwith the detection elements (e.g., detection elements 851A and 851B) maydetect first signals associated with a first area and second signalsassociated with a second area. Any of the processing of the first aridsecond signals described above with respect to FIG. 8A may similarly beperformed for the configuration of FIG. 8B.

In both FIGS. 8A and 8B, the detection elements may be integral to theshelf, part of a fabric or other surface configured for positioning onthe shelf, or the like. Power and/or data cables may form part of theshelf, the fabric, the surface, or be otherwise connected to thedetection elements. Additionally or alternatively, as depicted in FIGS.8A and 8B, individual sensors may be positioned on the shelf. Forexample, the power and/or data cables may be positioned under the shelfand connected through the shelf to the detection elements. In anotherexample, power and/or data may be transmitted wirelessly to thedetection elements (e.g., to wireless network interface controllersfanning part of the detection elements). In yet another example, thedetection elements may include internal power sources (such as batteriesor fuel cells).

With reference to FIG. 9 and consistent with the present disclosure, thedetection elements described above with reference to FIGS. 8A and 8B maybe arranged on rows of the shelf in any appropriate configuration. Allof the arrangements of FIG. 9 are shown as a top-down view of a row(e.g., area 805A, area 805B, area 855A, area 855B, or the like) on theshelf. For example, arrangements 910 and 940 are both uniformdistributions of detection elements within a row. However, arrangement910 is also uniform throughout the depth of the row while arrangement940 is staggered. Both arrangements may provide signals that representproducts on the shelf in accordance with spatially uniform measurementlocations. As further shown in FIG. 9, arrangements 920, 930, 950, and960 cluster detection elements near the front (e.g., a facing portion)of the row. Arrangement 920 includes detection elements at a frontportion while arrangement 930 includes defection elements in a largerportion of the front of the shelf. Such arrangements may save power andprocessing cycles by having fewer detection elements on a hack portionof the shelf. Arrangements 950 and 960 include some detection elementsin a hack portion of the shelf but these elements are arranged lessdense than detection elements in the front. Such arrangements may allowfor detections in the back of the shelf (e.g., a need to restockproducts, a disruption to products in the back by a customer oremployee, or the like) while still using less power and fewer processingcycles than arrangements 910 and 940. And, such arrangements may includea higher density of detection elements in regions of the shelf (e.g,, afront edge of the shelf) where product turnover rates may be higher thanin other regions (e.g., at areas deeper into a shelf), and/or in regionsof the shelf where planogram compliance is especially important.

FIG. 10A is a now chart, illustrating an exemplary method 1000 formonitoring planogram compliance on a store shelf, in accordance with thepresently disclosed subject matter. It is contemplated that method 1000may be used with any of the detection element arrays discussed abovewith reference to, for example, FIGS. 8A, 8B and 9. The order andarrangement of steps in method 1000 is provided for purposes ofillustration. As will be appreciated from this disclosure, modificationsmay be made to process 1000, for example, adding, combining, removing,and/or rearranging one or more steps of process 1000.

Method 1000 may include a step 1005 of receiving first signals from afirst subset of detection elements (e.g., detection elements 801A and801B of FIG. 8A) from among the plurality of detection elements afterone or more of a plurality of products (e.g., products 803A and 803B)are placed on at least one area of the store shelf associated with thefirst subset of detection elements. As explained above with respect toFIGS. 8A and 8B, the plurality of detection elements may be embeddedinto a fabric configured to be positioned on the store shelf.Additionally or alternatively, the plurality of detection elements maybe configured to be integrated with the store shelf. For example, anarray of pressure sensitive elements (or any other type of detector) maybe fabricated as part of the store shelf. In some examples, theplurality of detection elements may be configured to placed adjacent to(or located on) store shelves, as described above.

As described above with respect to arrangements 910 and 940 of FIG. 9,the plurality of detection elements may be substantially uniformlydistributed across the store shelf. Alternatively, as described abovewith respect to arrangements 920, 930, 950, and 960 of FIG. 9, theplurality of detection elements may be distributed relative to the storeshelf such that a first area of the store shelf has a higher density ofdetection elements than a second area of the store shelf. For example,the first area may comprise a front portion of the shelf, and the secondarea may comprise a hack portion of the shelf.

In some embodiments, such as those including pressure sensors or othercontact sensors as depicted in the example of FIG. 8A, step 1005 mayinclude receiving the first signals from the first subset of detectionelements as the plurality of products are placed above the first subsetof detection elements. In some embodiments where the plurality ofdetection elements includes pressure detectors, the first signals may beindicative of pressure levels detected by pressure detectorscorresponding to the first subset of detection elements after one ormore of the plurality of products are placed on the at least one area ofthe store shelf associated with the first subset of detection elements.For example, the first signals may be indicative of pressure levelsdetected by pressure detectors corresponding to the first subset ofdetection elements after stocking at least one additional product abovea product previously positioned on the shelf, removal of a product fromthe shelf; or the like. In other embodiments where the plurality ofdetection elements includes light detectors, the first signals may beindicative of light measurements made with respect to one or more of theplurality of products placed on the at least one area of the store shelfassociated with the first subset of detection elements. Specifically,the first signals may be indicative of at least part of the ambientlight being blocked from reaching the light detectors by the one or moreof the plurality of products.

In embodiments including proximity sensors as depicted in the example ofFIG. 8B, step 1005 may include receiving the first signals from thefirst subset of detection elements as the plurality of products areplaced below the first subset of detection elements. In embodimentswhere the plurality of detection elements include proximity detectors,the first signals may be indicative of proximity measurements made withrespect to one or more of the plurality of products placed on the atleast one area of the store shelf associated with the first subset ofdetection elements.

Method 1000 may include step 1010 of using the first signals to identifyat least one pattern associated with a product type of the plurality ofproducts. For example, any of the pattern matching techniques describedabove with respect to FIGS. 8A and 8B may be used for identification. Apattern associated with a product type may include a pattern (e.g., acontinuous ring, a discontinuous ring of a certain number of points, acertain shape, etc.) associated with a base of a single product. Thepattern associated with a product type may also be formed by a group ofproducts. For example, a six pack of soda cans may be associated with apattern including a 2×3 array of continuous rings associated with thesix carts of that product type. Additionally, a grouping of two literbottles may form a detectable pattern including an array (whetheruniform, irregular, or random) of discontinuous rings of pressurepoints, where the rings have a diameter associated with a particular2-liter product. Various other types of patterns may also be detected(e.g., patterns associated with different product types arrangedadjacent to one another, patterns associated with solid shapes (such asa rectangle of a boxed product), etc.). In another example, anartificial neural network configured to recognize product types may beused to analyze the signals received by step 1005 (such as signals frompressure sensors, from light detectors, from contact sensors, and soforth) to determine product types associated with products placed on anarea of a shelf (such as an area of a shelf associated with the firstsubset of detection elements). In yet another example, a machinelearning algorithm trained using training examples to recognize producttypes may be used to analyze the signals received by step 1005 (such assignals from pressure sensors, from light detectors, from contactsensors, and so forth) to determine product types associated withproducts placed on an area of a shelf (such as an area of a shelfassociated with the first subset of detection elements).

In some embodiments, step 1010 may further include accessing a memorystoring data (e.g., memory device 226 of FIG. 2 and/or memory device 314of FIG. 3A) associated with patterns of different types of products. Insuch embodiments, step 1010 may include using the first signals toidentify at least one product of a first type using a first pattern (ora first product model) and at least one product of a second type using asecond pattern (or a second product model). For example, the first typemay include one brand (such as Coca-Cola® or Folgers®) while the secondtype may include another brand (such as Pepsi® or Maxwell House®). Inthis example, a size, shape, point spacing, weight, resistance or otherproperty of the first brand may be different from that of the secondbrand such that the detection elements may differentiate the brands.Such characteristics may also be used to differentiate like-branded, butdifferent products from one another (e.g., a 12-ounce can of Coca Cola,versus a 16 oz bottle of Coca Cola, versus a 2-liter bottle of CocaCola). For example, a soda may have a base detectable by a pressuresensitive pad as a continuous ring. Further, the can of soda may beassociated with a first weight signal having a value recognizable asassociated with such a product. A 16 ounce bottle of soda may beassociated with a base having four or five pressure points, which apressure sensitive pad may detect as arranged in a pattern associatedwith a diameter typical of such a product. The 16 ounce bottle of sodamay also be associated with a second weight signal having a value higherthan the weight signal associated with the 12 ounce can of soda. Furtherstill, a 2 liter bottle of soda may be associated with a base having aring, four or five pressure points, etc, that a pressure sensitive padmay detect as arranged in a pattern associated with a diameter typicalof such a product. The 2 liter bottle of soda may be associated with aweight signal having a value higher than the weight signal associatedwith the 12 ounce can of soda and 16 ounce bottle of soda.

In the example of FIG. 8B, the different bottoms of product 853A andproduct 853B may be used to differentiate the products from each other.For example, detection elements such as pressure sensitive pads may beused to detect a product base shape and size (e.g., ring, pattern ofpoints, asymmetric shape, base dimensions, etc.). Such a base shape andsize may be used (optionally, together with one or more weight signals)to identify a particular product. The signals may also be used toidentify and/or distinguish product types from one another. For example,a first type may include one category of product (such as soda cans)while a second type may include a different category of product (such asnotepads). In another example, detection elements such as lightdetectors may be used to detect a product based on a pattern of lightreadings indicative of a product blocking at least part of the ambientlight from reaching the light detectors. Such pattern of light readingsmay be used to identify product type and/or product category and/orproduct shape. For example, products of a first type may block a firstsubset of light frequencies of the ambient light from reaching the lightdetectors, while products of a second type may block a second subset oflight frequencies of the ambient light from reaching the light detectors(the first subset and second subset may differ). In this case the typeof the products may be determined based on the light frequenciesreaching the light detectors. In another example, products of a firsttype may have a first shape of shades and therefore may block ambientlight from reaching light detectors arranged in one shape, whileproducts of a second type may have a second shape of shades andtherefore may block ambient light from reaching light detectors arrangedin another shape. In this case the type of the products may bedetermined based on the shape of blocked ambient light. Any of thepattern matching techniques described above may be used for theidentification.

Additionally or alternatively, step 1010 may include using the at leastone pattern to determine a number of products placed on the at least onearea of the store shelf associated with the first subset of detectionelements. For example, any of the pattern matching techniques describedabove may be used to identify the presence of one or more product typesand then to determine the number of products of each product type (e.g.,by detecting a number of similarly sized and shaped product bases andoptionally by detecting weight signals associated with each detectedbase), in another example, an artificial neural network configured todetermine the number of products of selected product types may be usedto analyze the signals received by step 1005 (such as signals frompressure sensors, from light detectors, from contact sensors, and soforth) to determine the number of products of selected product typesplaced on an area of a shelf (such as an area of a shelf associated withthe first subset of detection elements). In yet another example, amachine learning algorithm trained using training examples to determinethe number of products of selected product types may be used to analyzethe signals received by step 1005 (such as signals from pressuresensors, from light detectors, from contact sensors, and so forth) todetermine the number of products of selected product types placed on anarea of a shelf (such as an area of a shelf associated with the firstsubset of detection elements). Additionally or alternatively, step 1010may include extrapolating from a stored pattern associated with a singleproduct (or type of product) to determine the number of productsmatching the first signals. In such embodiments, step 1010 may, furtherinclude determining, for example based on product dimension data storedin a memory, a number of additional products that can be placed on theat least one area of the store shelf associated with the second subsetof detection elements. For example, step 1010 may include extrapolatingbased on stored dimensions of each product and stored dimensions of theshelf area to determine an area and/or volume available for additionalproducts. Step 1010 may further include extrapolation of the number ofadditional products based on the stored dimensions of each product anddetermined available area and/or volume.

Method 1000 may include step 1015 of receiving second signals from asecond subset of detection elements (e.g., detection elements 851A and851B of FIG. 8B) from among the plurality of detection elements, thesecond signals being indicative of no products being placed on at leastone area of the store shelf associated with the second subset ofdetection elements. Using this information, method 1000 may include step1020 of using the second signals to determine at least one empty spaceon the store shelf. For example, any of the pattern matching techniquesdescribed above may be used to determine that the second signals includedefault values or other values indicative of a lack of product incertain areas associated with a retail store shelf. A default value maybe include, for example, a pressure signal associated with an un-loadedpressure sensor or pressure sensitive mat, indicating that no product islocated in a certain region of a shelf. In another example, a defaultvalue may include signals from tight detectors corresponding to ambientlight, indicating that no product is located in a certain region of ashelf.

Method 1000 may include step 1025 of deter mining, based on the at leastone pattern associated with a detected product and the at least oneempty space, at least one aspect of planogram compliance. As explainedabove with respect to FIGS. 8A and 8B, the aspect of planogramcompliance may include the presence or absence of particular products(or brands), locations of products on the shelves, quantities ofproducts within particular areas (e.g., identifying stacked or clusteredproducts), facing directions associated with the products (e.g,, whethera product is outward facing, inward facing, askew, or the like), or thelike. A planogram compliance determination may be made, for example, bydetermining a number of empty spaces on a shelf and determining alocation of the empty spaces on a shelf. The planogram determination mayalso include determining weight signal magnitudes associated withdetected products at the various detected non-empty locations. Thisinformation may be used by the one or more processors in determiningwhether a product facing specification has been satisfied (e.g., whethera front edge of a shelf has a suitable number of products or suitabledensity of products), whether a specified stacking density has beenachieved (e.g., by determining a pattern of detected products and weightsignals of the detected products to determine how many products arestacked at each location), whether a product density specification hasbeen achieved (e.g., by determining a ratio of empty locations toproduct-present locations), whether products of a selected product typeare located in a selected area of the shelf, whether all productslocated in a selected area of the shelf are of a selected product type,whether a selected number of products (or a selected number of productsof a selected product type) are located in a selected area of the shelf,whether products located in a selected area of a shelf are positioned ina selected orientation, or whether any other aspect of one or moreplanograms has been achieved.

For example, the at least one aspect may include product homogeneity,and step 1025 may further include counting occurrences where a productof the second type is placed on an area of the store shelf associatedwith the first type of product. For example, by accessing a memoryincluding base patterns (or any other type of pattern associated withproduct types, such as product models), the at least one processor maydetect different products and product types. A product of a first typemay be recognized based on a first pattern, and product of a second typemay be recognized based on a second, different pattern (optionally alsobased on weight signal information to aid in differentiating betweenproducts). Such information may be used, for example, to monitor whethera certain region of a shelf includes an appropriate or intended productor product type. Such information may also be useful in determiningwhether products or product types have been mixed (e.g,, producthomogeneity). Regarding planogram compliance, detection of differentproducts and their relative locations on a shelf may aid in determiningwhether a product homogeneity value, ratio, etc. has been achieved. Forexample, the at least one processor may count occurrences where aproduct of a second type is placed on an area of the store shelfassociated with a product of a first type.

Additionally or alternatively, the at least one aspect of planogramcompliance may include a restocking rate, and step 1025 may furtherinclude determining the restocking rate based on a sensed rate at whichproducts are added to the at least one area of the store shelfassociated with the second subset of detection elements. Restocking ratemay be determined, for example, by monitoring a rate at which detectionelement signals change as products are added to a shelf (e.g., whenareas of a pressure sensitive pad change from a default value to aproduct-present value).

Additionally or alternatively, the at least one aspect of planogramcompliance may include product facing, and step 1025 may further includedetermining the product facing based on a number of products determinedto be placed on a selected area of the store shelf at a front of thestore shelf. Such product facing may be determined by determining anumber of products along a certain length of a front edge of a storeshelf and determining whether the number of products complies with, forexample, a specified density of products, a specified number ofproducts, and so forth.

Step 1025 may further include transmitting an indicator of the at leastone aspect of planogram compliance to a remote server. For example, asexplained above with respect to FIGS. 8A and 8B, the indicator maycomprise a data packet, a data file, or any other data structureindicating any variations from a planogram, e.g., with respect toproduct (or brand) placement, product facing direction, or the like. Theremote server may include one or more computers associated with a retailstore (e.g., so planogram compliance may be determined on a local basiswithin a particular store), one or more computers associated with aretail store evaluation body (e.g., so planogram compliance may bedetermined across a plurality of retail stores), one or more computersassociated with a product manufacturer, one or more computers associatedwith a supplier (such as supplier 115), one or more computers associatedwith a market research entity (such as market research entity 110), etc.

Method 1000 may further include additional steps. For example, method1000 may include identifying a change in at least one characteristicassociated with one or more of the first signals (e.g., signals from afirst group or type of detection elements), and in response to theidentified change, triggering an acquisition of at least one image ofthe store shelf. The acquisition may be implemented by activating one ormore of capturing devices 125 of FIGS. 4A-4C, as explained above. Forexample, the change in at least one characteristic associated with oneor more of the first signals may be indicative of removal of at leastone product from a location associated with the at least one area of thestore shelf associated with the first subset of detection elements.Accordingly, method 1000 may include triggering the acquisition todetermine whether restocking, reorganizing, or other intervention isrequired, e.g., to improve planogram compliance. Thus, method 1000 mayinclude identifying a change in at least one characteristic associatedwith one or more of the first signals; and in response to the identifiedchange, trigger a product-related task for an employee of the retailstore.

Additionally or alternatively, method 1000 may be combined with method1050 of FIG. 10B, described below, such that step 1055 is performed anytime after step 1005.

FIG. 10B is a flow chart, illustrating an exemplary method 1050 fortriggering image capture of a store shelf, in accordance with thepresently disclosed subject matter. It is contemplated that method 1050may be used in conjunction with any of the detection element arraysdiscussed above with reference to, for example, FIGS. 8A, 8B and 9. Theorder and arrangement of steps in method 1050 is provided for purposesof illustration. As will be appreciated from this disclosure,modifications may be made to process 1050, for example, adding,combining, removing, and/or rearranging one or more steps of process1050.

Method 1050 may include a step 1055 of determining a change in at leastone characteristic associated with one or more first signals. Forexample, the first signals may have been captured as part of method 1000of FIG. 10A, described above. For example, the first signals may includepressure readings when the plurality of detection elements includespressure sensors, contact information when the plurality of detectionelements includes contact sensors, light readings when the plurality ofdetection elements includes light detectors (for example, from lightdetectors configured to be placed adjacent to (or located on) a surfaceof a store shelf configured to hold products, as described above), andso forth.

Method 1050 may include step 1060 of using the first signals to identifyat least one pattern associated with a product type of the plurality ofproducts. For example, any of the pattern matching techniques describedabove with respect to FIGS. 8A, 8B, and step 1010 may be used foridentification.

Method 1050 may include step 1065 of determining a type of eventassociated with the change. For example, a type of event may include aproduct removal, a product placement, movement of a product, or thelike.

Method 1050 may include step 1070 of triggering an acquisition of atleast one image of the store shelf when the change is associated with afirst event type. For example, a first event type may include removal ofa product, moving of a product, or the like, such that the first eventtype may trigger a product-related task for an employee of the retailstore depending on analysis of the at least one image. The acquisitionmay be implemented by activating one or more of capturing devices 125 ofFIGS. 4A-4C, as explained above. In some examples, the triggeredacquisition may include an activation of at least one projector (such asprojector 632). In some examples, the triggered acquisition may includeacquisition of color images, depth images, stereo images, active stereoimages, time of flight images, LIDAR images, RADAR images, and so forth.

Method 1050 may include a step (not shown) of forgoing the acquisitionof at least one image of the store shelf when the change is associatedwith a second event type. For example, a second event type may includereplacement of a removed product by a customer, stocking of a shelf byan employee, or the like. As another example, a second event type mayinclude removal, placement, or movement of a product that is detectedwithin a margin of error of the detection elements and,/or detectedwithin a threshold (e.g., removal of only one or two products: movementof a product by less than 5 cm, 20 cm, or the like; moving of a facingdirection by less than 10 degrees; or the like), such that no imageacquisition is required.

FIGS. 11A-11E illustrate example outputs based on data automaticallyderived from machine processing and analysis of images captured inretail store 105 according to disclosed embodiments. FIG. 11Aillustrates an optional output for market research entity 110. FIG. 11Billustrates an optional output for supplier 115. FIGS. 11C and 11Dillustrate optional outputs for employees of retail store 105. And FIG.11E illustrates optional outputs for user 120.

FIG. 11A illustrates an example graphical user interface (GUI) 500 foroutput device 145A, representative of a GUI that may be used by marketresearch entity 110. Consistent with the present disclosure, marketresearch entity 110 may assist supplier 115 and other stakeholders inidentifying emerging trends, launching new products, and/or developingmerchandising and distribution plans across a large number of retailstores 105. By doing so, market research entity 110 may assist supplier315 in growing product presence and maximizing or increasing new productsales. As mentioned above, market research entity 110 may be separatedfrom or part of supplier 115. To successfully launch a new product,supplier 115 may use information about what really happens in retailstore 105. For example, supplier 115 may want to monitor how marketingplans are being executed and to learn what other competitors are doingrelative to certain products or product types. Embodiments of thepresent disclosure may allow market research entity 110 and suppliers115 to continuously monitor product-related activities at retail stores105 (e.g., using system 100 to generate various metrics or informationbased on automated analysis of actual, timely images acquired from theretail stores). For example, in some embodiments, market research entity110 may track how quickly or at what rate new products are introduced toretail store shelves, identify new products introduced by variousentities, assess a supplier's brand presence across different retailstores 105, among many other potential metrics.

In some embodiments, server 135 may provide market research entity 110with information including shelf organization, analysis of skewproductivity trends, and various reports aggregating information onproducts appearing across large numbers of retail stores 105. Forexample, as shown in FIG. 11A, GUI 1100 may include a first, displayarea 1102 for showing a percentage of promotion campaign compliance indifferent retail stores 105. GUI 1100 may also include a second displayarea 1104 showing a graph illustrating sales of a certain productrelative to the percentage of out of shelf. GUI 1100 may also include athird display area 1106 showing actual measurements of different factorsrelative to target goals (e.g., planogram compliance, restocking rate,price compliance, and other metrics). The provided information mayenable market research entity 110 to give supplier 115 informed shelvingrecommendations and fine-tune promotional strategies according toin-store marketing trends, to provide store managers with a comparisonof store performances in comparison to a group of retail stores 105 orindustry wide performances, and so forth.

FIG. 11B illustrates an example GUI 1110 for output device 1451 used bysupplier 115, Consistent with the present disclosure, server 135 may usedata derived from images captured in a plurality of retail stores 105 torecommend a planogram, which often determines sales success of differentproducts. Using various analytics and planogram productivity measures,server 135 may help supplier 115 to determine an effective planogramwith assurances that most if not all retail stores 105 can execute theplan. For example, the determined planogram may increase the probabilitythat inventory is available for each retail store 105 and may bedesigned to decrease costs or to keep costs within a budget (such asinventory costs, restocking costs, shelf space costs, etc.). Server 135may also provide pricing recommendations based on the goals of supplier115 and other factors. In other words, server 135 may help supplier 115understand how much room to reserve for different products and how tomake them available for favorable sales and profit impact (for example,by choosing the size of the shelf dedicated to a selected product, thelocation of the shelf, the height of the shelf, the neighboringproducts, and so forth). In addition, server 135 may monitor nearreal-time data from retail stores 105 to determine or confirm thatretail stores 105 are compliant with the determined planogram ofsupplier 115. As used herein, the term “near real-time data,” in thecontext of this disclosure, refers to data acquired or generated, etc.,based on sensor readings and other inputs (such as data from imagesensors, audio sensors, pressure sensors, checkout stations, etc.) fromretail store 105 received by system 100 within a predefined period oftime (such as time periods having durations of less than a second, lessthan a minute, less than an hour, less than a day, less than a week,etc.).

In some embodiments, server 135 may generate reports that summarizeperformance of the current assortment and the planogram compliance.These reports may advise supplier 115 of the category and the itemperformance based on individual SKU, sub segments of the category,vendor, and region. In addition, server 135 may provide suggestions orinformation upon which decisions may be made regarding how or when toremove markdowns and when to replace underperforming products. Forexample, as shown in FIG. 11B, GUI 1110 may include a first display area1112 for showing different scores of supplier 115 relative to scoresassociated with its competitors. GUI 1110 may also include a seconddisplay area 1114 showing the market share of each competitor. GUI 1110may also include a third display area 1116 showing retail measurementsand distribution of brands. GUI 1110 may also include a fourth displayarea 1118 showing a suggested planogram. The provided information mayhelp supplier 115 to select preferred planograms based on projected orobserved profitability, etc., and to ensure that retail stores 105 arefollowing the determined planogram.

FIGS. 11C and 11D illustrate example GUIs for output devices 145C, whichmay be used by employees of retail store 105. FIG. 11C depicts a GUI1120 for a manager of retail store 105 designed for a desktop computer,and FIG. 11D depicts GUI 1130 and 1140 for store staff designed for ahandheld device. In-store execution is one of the challenges retailstores 105 have in creating a positive customer experience. Typicalin-store execution may involve dealing with ongoing service events, suchas a cleaning event, a restocking event, a rearrangement event, andmore. In some embodiments, system 100 may improve in-store execution byproviding adequate visibility to ensure that the right products arelocated at preferred locations on the shelf. For example, using nearreal-time data (e.g., captured images of store shelves) server 135 maygenerate customized online reports. Store managers and regionalmanagers, as well as other stakeholders, may access custom dashboardsand online reports to see how in-store conditions (such as, planogramcompliance, promotion compliance, price compliance, etc.) are affectingsales. This way, system 100 may enable managers of retail stores 105 tostay on top of burning issues across the floor and assign employees toaddress issues that may negatively impact the customer experience.

In some embodiments, server 135 may cause real-time automated alertswhen products are out of shelf (or near out of shelf), when pricing isinaccurate, when intended promotions are absent, and/or when there areissues with planogram compliance, among others. In the example shown inFIG. 11C, GUI 1120 may include a first display area 1122 for showing theaverage scores (for certain metrics) of a specific retail store 105 overa selected period of time. GUI 1120 may also include a second displayarea 1124 for showing a map of the specific retail store 105 withreal-time indications of selected in-store execution events that requireattention, and a third display area 1126 for showing a list of theselected in-store execution events that require attention. In anotherexample, shown in FIG. 11D, GUI 1130 may include a first display area1132 for showing a list of notifications or text messages indicatingselected in-store image (or the image itself) of the specific aisle withthe in-store execution event. In another example, shown in FIG. 11D, GUI1140 may include a first display area 1142 for showing a display of avideo stream captured by output device 145C (e.g., a real-time displayor a near real-time display) with augmented markings indicting a statusof planogram compliance for each product (e.g., correct place,misplaced, not in planogram, empty, and so forth). GUI 1140 may alsoinclude a second display area 1144 for showing a summary of theplanogram compliance for all the products identified in the video streamcaptured by output device 145C. Consistent with the present disclosure,server 135 may generate within minutes actionable tasks to improve storeexecution. These tasks may help employees of retail store 105 to quicklyaddress situations that can negatively impact revenue and customerexperience in the retail store 105.

FIG. 11E illustrates an example GUI 1150 for output device 145D used byan online customer of retail store 105. Traditional online shoppingsystems present online customers with a list of products. Productsselected for purchase may be placed into a virtual shopping cart untilthe customers complete their virtual shopping trip. Virtual shoppingcarts may be examined at any time, and their contents can be edited ordeleted. However, common problems of traditional online shopping systemsarise when the list of products on the website does not correspond withthe actual products on the shelf. For example, an online customer mayorder a favorite cookie brand without knowing that the cookie brand isout-of-stock. Consistent with some embodiments, system 100 may use imagedata acquired by capturing devices 125 to provide the online customerwith a near real-time display of the retail store and a list of theactual products on the shelf based on near real-time data. In oneembodiment, server 135 may select images without occlusions in the fieldof view (e.g., without other customers, carts, etc.) for the nearreal-time display. In one embodiment, server 135 may blur or erasedepictions of customers and other people from the near real-timedisplay. As used herein, the term “near real-time display,” in thecontext of this disclosure, refers to image data captured in retailstore 105 that was obtained by system 100 within a predefined period oftime (such as less than a second, less than a minute, less than about 30minutes, less than an hour, less than 3 hours, or less than 12 hours)from the time the image data was captured.

Consistent with the present disclosure, the near real-time display ofretail store 105 may be presented to the online customer in a mannerenabling easy virtual navigation in retail store 105. For example, asshown in FIG. 11E, GUI 1150 may include a first display area 1152 forshowing the near real-time display and a second display area 1154 forshowing a product list including products identified in the nearreal-time display. In some embodiments, first display area 1152 mayinclude different GUI features (e.g., tabs 1156) associated withdifferent locations or departments of retail store 105. By selectingeach of the GUI features, the online customer can virtually jump todifferent locations or departments in retail store 105. For example,upon selecting the “bakery” tab, GUI 1150 may present a near real-timedisplay of the bakery of retail store 105. In addition, first displayarea 1152 may include one or more navigational features (e.g., arrows1158A and 1158B) for enabling the online customer to virtually movewithin a selected department and/or virtually walk through retail store105. Server 135 may be configured to update the near real-time displayand the product list upon determining that the online customer wants tovirtually move within retail store 105. For example, after identifying aselection of arrow 1158B, server 135 may present a different section ofthe dairy department and may update the product list accordingly. Inanother example, server 135 may update the near-real time display andthe product list in response to new captured images and new informationreceived from retail store 105. Using GUI 1150, the online customer mayhave the closest shopping experience without actually being in retailstore 105. For example, an online customer can visit the vegetabledepartment and decide not to buy tomatoes after seeing that they are notripe enough.

Embodiments of the present disclosure may allow for a system forproviding incentives to individuals (such as non-employee individuals,employee individuals, etc.) to complete tasks associated with a retailstore. For example, the system may receive an image of a store shelf inthe store. The system may analyze the image and determine that the storeshelf does not comply with a planogram (e.g., there is an expiredpromotional sign attached to the store shelf). The system may alsodetermine a task associated with the planogram incompliance (e.g.,removing the expired promotional sign). The system may further providean offer to an individual (for example to a non-employee individual,such as a customer in the store, etc.) for completing the task inexchange for a reward.

FIG. 12 is a schematic illustration of an example system 1200 forproviding offers to non-employee individuals for completing tasksassociated with a retail store in exchange for rewards, consistent withthe present disclosure. As illustrated in FIG. 12, system 1200 mayinclude a computing device 1201, database 1202, one or more user devices1204A, 1204B, 1204C, . . . , 1204N (or user device 1204, collectively)associated with non-employee individuals 1203A, 1203B, 1203C, . . . ,1203N, and database 1202.

Computing device 1201 may be configured to analyze one or more images todetermine a task associated with a store shelf. Computing device 1201may also be configured to determine an incentive for an individual (suchas a non-employee individual, employee individual, etc.) to complete thetask. A non-employee individual referred to herein is an individual whois not employed by the retail store. It is to be understood thatalthough non-employees are used as example individuals, the systems andmethods disclosed herein are not limited to non-employees, and may beequally implemented for other individuals, such as employees of theretail stores. Computing device 1201 may further be configured toprovide an offer to the non-employee individual to complete the task andprovide the incentive to the non-employee individual if be or shecompletes the task. In some embodiments, computing device 1201 mayinclude image processing unit 130.

Database 1202 may be configured to store information and/or data for oneor more components of system 1200. For example, database 1202 may beconfigured to store one or more images captured by capturing device 125.Computing device 1201 may access the one or more images stored indatabase 1202. As another example, database 1202 may store informationrelating to a retail store, information relating to one or more storeshelves of a retail store, information relating to one or morenon-employee individuals, or the like, or a combination thereof.Computing device 1201 may access information stored in database 1202.For example, computing device 1201 may access information relating toone or more properties of a store shelf, the physical location of thestore shelves, one or more products associated with the store shelf.

Network 1205 may be configured to facilitate the communication betweenthe components of system 1200. For example, computing device 1201 maysend to arid receive from user device 1204 information and/or data vianetwork 1205. Network 1205 may include a local area network (LAN), awide area network (WAN), portions of the Internet, an Intranet, acellular network, a short-ranged network (e.g,, a Bluetooth™ basednetwork), or the like, or a combination thereof.

User device 1204 may be configured to receive information and/or datafrom computing device 1201. For example, user device 1204 may beconfigured to receive a notification indicating an offer from computingdevice 1201 for completing a task associated with a store shelf. Userdevice 1204 may include an output device (e.g., a display) configured toprovide the user with a graphical user interface (GUI) including thenotification. User device 1204 may also include an input device (e.g., atouch screen, button, etc.) configured to receive input from the user.For example, user device 1204 may receive user input from the user viathe input device indicating that the user accepts the offer receivedfrom computing device 1201.

FIGS. 13A and 13B illustrate an example GUI 1310 displayed on userdevice 1204, which may be used by a non-employee individual of retailstore 105. User device 1204 may receive a notification representing anoffer for the non-employee individual associated with user device 1204to complete a task from computing device 1201. User device 1204 maydisplay the notification relating to the offer in GUI 1310. Thenotification may ask the non-employee individual associated with theuser device 1301 whether he or she would accept an offer to receive acoupon (e.g., a coupon of $10) in exchange for completing a task. GUI1310 may also be configured to receive user input from the non-employeeindividual indicating the non-employee individual's acceptance ordecline of the offer. In some embodiments. GUI 1310 may also includemore detailed information relating to the incentive and/or task. In someembodiments, if the non-employee individual accepts the offer (by, e.g.,selecting the “YES” button illustrated in FIG. 13A on a touch screen),and GUI 1310 may display information relating to the performance of thetask. For example, as illustrated in FIG. 13B, GUI 1310 may display mayinclude a first display area 1311 for showing a display of an image ofthe store shelf (e.g., a real-time display or a near real-time displaycaptured by a capturing device) with augmented markings indicting astatus of planogram compliance for each product (e.g., correct place,misplaced, not in planogram, empty, and so forth). In some embodiments,GUI 1310 may also include a second display area 1312 for showing asummary of the planogram compliance for all the products identified inthe image. The information displayed in GUI 1310 may help thenon-employee individual complete the task (e.g., rearranging theproducts placed on the store shelf, restocking certain products, etc.).Computing device 1201 may further determine whether the task has beencompleted. For example, computing device 1201 may obtain one or moreimages of the store shelf (by, e.g., requesting the non-employeeindividual capture and upload an image of the store shelf) and determinewhether the task is completed based on the analysis of the one or moreimages. If computing device 1201 determines that the task is completed,computing device 1201 may provide the reward to the non-employeeindividual. For example, computing device 1201 may transmit a digitalcoupon corresponding to the incentive to user device 1301, to anotherdevice of the individual, and/or to an email address associated withindividual.

FIG. 14 is a flowchart representing an exemplary method 1400 fordetermining tasks for non-employee individuals in retail store 105 inaccordance with example embodiments of the present disclosure. Forpurposes of illustration, in the following description, reference ismade to certain components of system 1200 as deployed in theconfiguration depicted in FIG. 12. For example, one or more steps ofmethod 1400 may be performed by computing device 1201. It will beappreciated, however, that other implementations are possible and thatother configurations may be utilized to implement the exemplary method.It will also be readily appreciated that the illustrated method can bealtered to modify the order of steps, delete steps, or further includeadditional steps.

At step 1401, the method may include receiving one or more imagescaptured in a retail store and depicting a plurality of productsdisplayed on at least one store shelf.

For example, computing device 1201 may receive one or more imagescaptured in a retail store by one or more capturing device 125 asdescribed elsewhere in this disclosure. By way of example, computingdevice 1201 may receive one or more images captured in the retail storeby a handheld device of a store employee (e.g., capturing device 125D),which may depict a shelf and one or more products placed thereon.Alternatively or additionally, computing device 1201 may receive one ormore images captured by crowd sourcing (e.g., by user devices 1204A-Nassociated with non-employee individuals in the retail store), asdescribed elsewhere in this disclosure. Alternatively or additionally,computing device 1201 may receive one or more images captured by camerasmounted in the retail store, as described elsewhere in this disclosure.

At step 1403, the method may include analyzing the one or more images todetermine at least one task associated with a store shelf. For example,computing device 1201 may analyze the one or more images and determine atask associated with a store shelf based on the image analysis.Exemplary tasks associated with a store shelf may include at least oneof capturing at least one image of the store shelf, removing a labelassociated with the store shelf, attaching a label to the store shelf,modifying a label associated with the store shelf, removing apromotional sign associated with the store shelf, placing a promotionalsign next to the store shelf, modifying a promotional sign associatedwith the store shelf, entering information related to products placed onthe store shelf (such as name of a product, type of a product, categoryof a product, size of a product, brand of a product, amount of products,etc.), removing a product from the store shelf (for example, returning amisplaced product to a correct shelf, moving the product to anothershelf, moving the product to a stockroom, etc.), placing a product onthe store shelf (for example, returning a misplaced product from anincorrect shelf, moving the product from another shelf, moving theproduct from a stockroom, etc.), and so forth.

For example, a machine teaming model may be trained using trainingexamples to determine tasks associated with store shelves from imagesand/or videos, and step 1403 may use the trained machine learning modelto analyze the one or more images received using step 1401 to determinethe at least one task associated with a store shelf An example of suchtraining may include an image and/or a video of a store shelf, togetherwith a label indicating one or more tasks associated with the storeshelf. In some examples, computing device 1201 may analyze the one ormore images depicting a store shelf to determine planogram compliancerelative to the store shelf, as described elsewhere in this disclosure.By way of example, computing device 1201 may analyze an image of a storeshelf and determine at least one characteristic of planogram compliancebased on detected differences between the at least one planogram and theactual placement of the products on the store shelf based on the imageanalysis. Exemplary characteristics of planogram compliance may includeat least one of product facing, product placement, planogramcompatibility, price correlation, promotion execution, producthomogeneity, restocking rate, and planogram compliance of adjacentproducts. Computing device 1201 may also determine a task of rearrangingthe products on the store shelf to meet the planogram compliance basedon the characteristic of planogram compliance.

A task may include a task relating to an area of the retail store (e.g.,a store shelf). For example, a task may include taking one or moreadditional images of a store shelf, providing information relating tothe store shelf, and changing the position of the store shelf (e.g., theheight of the store shelf), By way of example, computing device 1201 maydetermine that the one or more images relating to a store shelf are outof date (e.g., the time since the last image of the store shelf exceedsa threshold) based on the analysis of the image data and determine atask for taking one or more additional images of the store shelf.

Alternatively or additionally, a task may include a task relating to oneor more characteristics of planogram compliance. For example, computingdevice 1201 may analyze the one or more images to determine at least onetask for maintaining planogram compliance, which may be based on atleast one of: product facing, product placement, planogramcompatibility, price correlation, promotion execution, and producthomogeneity, restocking rate, planogram compliance of adjacent products,or the like, or a combination thereof. By way of example, computingdevice 1201 may analyze the one or more images and determine a task formaintaining planogram compliance of the store shelf including at leastone of removing a product from the store shelf, removing a labelattached to with the store shelf, removing a promotional sign associatedwith the store shelf, or rearranging products to change product facing,or the like, or a combination thereof. Alternatively or additionally,computing device 1201 may analyze the one or more images and identify amisplaced product on the store shelf in the one or more images.Computing device 1201 may also determine a task for removing themisplaced product from the store shelf. Alternatively or additionally,computing device 1201 may determine a task for placing (or returning)the misplaced product to another store shelf (e.g., a store shelf remotefrom the store shelf associated with the one or more images).Alternatively or additionally, computing device 1201 may identify anexpired promotional sign associated with the store shelf in the one ormore images, and determine a task for removing the expired promotionalsign. Alternatively or additionally, computing device 1201 may identifyan unrecognized product placed on the store shelf identified in the oneor more images, and determine a task for providing information regardingthe unrecognized product. Alternatively or additionally, computingdevice 1201 may identify an incorrect label attached to the at least onestore shelf in the one or more images, and determine a task for removingthe incorrect label. Alternatively or additionally, computing device1201 may determine that a restocking of one or more products associatedwith a store shelf based on the image analysis is needed, and determinea task for restocking the store shelf with products from a particularstorage area.

In some embodiments, a task may be determined based on a comparisonbetween a planogram and one or more images. For example, computingdevice 1201 may analyze the one or more images to determine ant actualplacement of products on the store shelf. Computing device 1201 may alsoaccess at least one planogram describing a desired placement of productson the store shelf and determine a task based on detected differencesbetween the desired placement and the actual placement of products onthe store shelf Alternatively or additionally, computing device 1201 mayaccess at least one image depicting previous placement of products onthe store shelf, and determine the task based on detected differencesbetween the previous placement and the current placement of products onthe store shelf.

In some embodiments, computing device 1201 may receive data related to atask associated with a store shelf. For example, computing device 1201may receive the data related to the task from memory, a database (e.g.,database 1202), a blockchain, an external device over a communicationnetwork using a communication device, or the like, or a combinationthereof.

At step 1405, the method may include determining an incentive forcompleting the at least one task based on a property of the store shelf,wherein the incentive is intended for a non-employee individualassociated with the retail store. For example, computing device 1201 maydetermine an incentive for a non-employee individual (e.g., a customerin the retail store) for completing the task determined at step 1403.Exemplary incentives may include at least one of a discount, a monetaryincentive, a coupon, an award, a free product, or the like, or acombination thereof. By way of example, computing device 1201 maydetermine a coupon of $10 to be used in the retail store for rearrangingthe products on a store shelf according to a planogram. In someexamples, a machine learning model may be trained using trainingexamples to determine incentives for completing tasks associated withstore shelves from properties of the store shelves, and step 1405 mayuse the trained machine learning model to analyze the property of thestore shelf and determine the incentive for completing the at least onetask. An example of such training example may include properties of astore shelf and/or an indication of a task to be completed, togetherwith a label indicating a desired incentive for completing the task. Insome examples, a machine learning model may be trained using trainingexamples to determine incentives for completing tasks associated withstore shelves From images and/or videos of the store shelves, and step1405 may use the trained machine learning model to analyze the one ormore images received using step 1401 and determine the incentive forcompleting the at least one task. An example of such training examplemay include an image and/or a video of a store shelf, for example withan indication of a task to be completed, together with a labelindicating a desired incentive for completing the task.

In some embodiments, an incentive may be determined based on a propertyof the store shelf, type of products placed on the store shelf, aproperty of the storage area for storing one or more products associatedwith the store shelf, emergency level of the task, or the like, or acombination thereof. For example, computing device 1201 may determine anincentive for completing a task associated with a store shelf (forexample, for completing a task for maintaining planogram compliance ofthe store shelf) and determine an incentive for completing the taskbased on the property of the store shelf associated with a physicallocation of the store shelf (e.g., a remote area in the retail store,being close to or far away from the entrance and/or exit of the retailstore, being close to or far away from the non-employee individual,being at an eye level of an individual, the top or the bottom of thestore shelf, or the like, or a combination thereof). As another example,the task may include a task for returning a misplaced product from thestore shelf to a remote store shelf. Computing device 1201 may determinean incentive for completing the task based on a property of the storeshelf and/or a property of the remote store shelf. By way of example,computing device 1201 may determine a type of the misplaced productbased on, for example, the analysis one or more images of the misplacedproduct. Computing device 1201 may also determine a remote store shelfon which the misplaced product should be placed based on the type of themisplaced product. Computing device 1201 may further determine anincentive for completing the task (including removing the misplacedproducts from the previous store shelf and placing the misplaced producton the remote store shelf) based on a property of the remote store shelf(e.g., a distance from the previous store shelf on which the misplacedproduct is placed) and/or the property of the previous store shelf. Inone example, properties of the remote store shelf may be determined byanalyzing images of the remote store shelf (for example, as describedherein with respect to the store shelf). In one example, properties ofthe remote store shelf may be received from a data structure includinginformation related to store shelves, such as a floor plan of the retailstore, a database of the retail store, and so forth.

In some embodiments, an incentive may be determined based on a propertyof the store shelf. Exemplary properties of a store shelf may includeinformation related to at least one of products placed on the at leastone store shelf, empty spaces on the store shelf, a label attached tothe store shelf, a promotional sign associated with the store shelf, aheight of the store shelf, a type of the store shelf, a location of thestore shelf within the retail store, a rate of customers passing in avicinity of the store shelf, or the like, or a combination thereof. Forexample, any of the properties mentioned above may be used to determinethe incentive for the task. Some possible examples of such incentivesmay include at least one: of a discount, a monetary incentive, a coupon,an award, a free product, and so forth.

In some example, the at least one property of the at least one storeshelf may be used to select the at least one user of a plurality ofalternative users. For example, any of the properties mentioned abovemay be used to select the at least one user of the plurality ofalternative users. For example, users compatibility to shelves withdifferent properties may be obtained (for example, based on pastperformances, based on past training, based on past responsiveness,etc.), and the obtained compatibility to shelves may be used to selectthe more compatible users to the at least one property of the at leastone store shelf from the plurality of alternative users as the at leastone user. In another example, a machine learning model may be trainedusing training examples to select users based on properties of shelves,and the trained machine learning model may be used to select the atleast one user based on the at least one property of the at least onestore shelf: In yet another example, an artificial neural network may beconfigured to select users according to properties of shelves, and theartificial neural network may be used to select the at least one useraccording to the at least one property of the at least one store shelf.

In some embodiments, the property of the store shelf may be determinedbased on an analysis of the one or more images. For example, computingdevice 1201 may analyze the one or more images and determine thelocation of the store shelf based on the image analysis. For example, amachine learning model trained using training examples to identify oneor more properties of store shelves from images and/or videos of thestore shelf, and the image data may be analyzed using the trainedmachine learning model to identify one or more properties of a storeshelf. An example of such training example may include an image and/or avideo of a store shelf, together with a label indicating one or moreproperties of the store shelf. Alternatively or additionally, anartificial neural network may be configured to identify one or moreproperties of a store shelf from images and/or videos of the storeshelf, and the image data may be analyzed using the artificial neuralnetwork to identify one or more properties. Alternatively oradditionally, computing device 1201 may receive information relating tothe property of the store shelf from memory, a database (e.g., database1202), a blockchain, an external device over a communication networkusing a communication device, or the like, or a combination thereof.Alternatively or additionally, computing device 1201 may determine theproperty of the store shelf based on the information obtained fromanother sensor (e.g., indoor positioning sensors).

Alternatively or additionally, an incentive may be determined based on aproperty of the store shelf associated with a type of products placed onthe store shelf. For example, computing device 3201 may determine a moresignificant incentive for completing a task associated with a type ofproducts that may be more difficult to handle. By way of example,computing device 1201 may determine a more significant incentive forcompleting a task associated with frozen food than for completing tasksassociated with canned food. Alternatively or additionally, computingdevice 1201 may determine a more significant incentive for completingtasks associated with heavy products (e.g., a product having a weightheavier than 2 kilograms) than for completing tasks associated withlight products (e.g., a product having a weight lighter than 1kilogram).

Alternatively or additionally, an incentive may be determined based on aproperty of the storage area for storing one or more products associatedwith the store shelf. For example, the task may include a task forrestocking the store shell with products from a particular storage area,and computing device 1201 may determine an incentive for completing thetask based on a property of the particular storage area. Exemplaryproperties of the particular storage area may include a distance of thestore shelf from the particular storage area, a height of the particularstorage area, a location of the particular storage area, or the like, ora combination thereof. In one example, properties of the storage areamay be determined by analyzing images of the storage area (for example,to determine properties of the storage area, to determine inventorylevels of the storage area, using a machine learning model, using anartificial neural network, and so forth). In one example, properties ofthe storage area may be received from a data structure includinginformation related to storage areas, such as a floor plan of the retailstore, a database of the retail store, and so forth.

Alternatively or additionally, an incentive may be determined based onan importance or emergency level of the task. For example, computingdevice 1201 may determine the importance or emergency level of a taskbased on a characteristic of the task. By way of example, computingdevice 1201 may determine a low (or non) urgent level for restockingmostly full shelf, placing promotion sign, and/or relocating of amisplaced product (non-frozen). Alternatively or additionally, computingdevice 1201 may determine a medium urgency level for a task for productfacing. Alternatively or additionally, computing device 1201 maydetermine a high urgency level for a task for correcting an incorrectlabel, restocking empty shelf, relocating frozen or refrigeratedproducts that have been placed in a non-frozen or non-refrigerated area,or the like, or a combination thereof. Computing device 1201 may alsodetermine a more significant incentive for completing urgent tasks thanto complete less urgent tasks.

In some embodiments, computing device 1201 may determine a moresignificant (or less significant) incentive based on a property of thestore shelf, type of products placed on the store shelf, a property ofthe storage area for storing one or more products associated with thestore shelf, emergency level of the task, or the like, or a combinationthereof. For example, computing device 1201 may determine a moresignificant incentive for completing a task that is associated withstore shelves located in less-visited areas of the retail store than forcompleting similar tasks associated with store shelves located inwell-visited areas. As another example, computing device 1201 maydetermine an incentive with less value (e.g., a coupon having a lessdiscount) for completing a task that is associated with shelves close toone or more cashiers. Alternatively or additionally, computing device1201 may determine a more significant incentive for completing tasksassociated with bottom (or top) store shelves than for completingsimilar tasks associated with store shelves located at eye level.

In some embodiments, the incentive may be determined based on one ormore models, which may include one or more machine learning models, oneor more regression models, one or more artificial neural networks, orthe like, or a combination thereof. For example, computing device 1201may determine an amount associated with the incentive for the task fromthe property of the store shelf using a regression model. Alternativelyor additionally, computing device 1201 may determine an incentive forthe task based on a training deep learning model. By way of example,computing device 1201 may obtain a deep learning model for determiningan incentive that has been trained based on a plurality of trainingsamples. The deep learning model may have been trained using a pluralityof positive training samples (e.g., samples for which an offer having anincentive for completing a task was accepted by a non-employeeindividual) and/or a plurality of negative training samples (e.g.,samples for which an offer of an incentive for completing a task was notaccepted by a non-employee individual). Computing device 1201 may alsoinput the data relating to the task and/or the property of the retailstore into the deep learning model, which may output the results fordetermining an incentive (e.g., the type, value, or the like, or acombination thereof). Alternatively or additionally, computing device1201 may use an artificial neural network may be used to determine theincentive for the task according to the property of the store shelf.

At step 1407, the method may include providing an offer to at least oneindividual non-employee individuals associated with the retail store.The offer may include receiving the incentive in response to completingthe at least one task.

For example, computing device 1201 may generate an offer to anon-employee individual for completing a task in exchange for receivingthe incentive determined at step 1405. Computing device 1201 may alsoprovide the offer to a user device associated with the non-employeeindividual. By way of example, computing device 1201 may provide anon-employee individual an offer associated with a task of removing amisplaced product (if a misplaced product has been identified), removingan expired promotional sign (if an expired promotional sign has beenidentified), providing information regarding an unrecognized product (ifan unrecognized product has been identified), removing an incorrectlabel (if an incorrect label has been identified), capturing anadditional image of the store shelf (if the one or more images aredetermined to be out of date), or the like, or a combination thereof.

In some embodiments, computing device 1201 may transmit (or push) anotification representing an offer to user device 1301 (as illustratedin FIG. 13A), which may be displayed in GUI 1310 of user device 1301.The notification may ask the non-employee individual associated with theuser device 1301 whether he or she would accept an offer to receive acoupon of $10 in exchange for completing a task, GUI 1310 may also beconfigured to receive user input from the non-employee individualindicating the non-employee individual's acceptance or decline of theoffer. In some embodiments, GUI 1310 may also include more detailedinformation relating to the incentive and/or task. In some embodiments,if the non-employee individual accepts the offer (by, e.g., selectingthe “YES” button illustrated in FIG. 13A), and GUI 1310 may displayinformation relating to the performance of the task. For example, asillustrated in FIG. 13B, GUI 1310 may display may include a firstdisplay area 1311 for showing a display of an image of the store shelf(e.g., a real-time display or a near real-time display captured by acapturing device) with augmented markings indicting a status ofplanogram compliance for each product (e.g., correct place, misplaced,not in planogram, empty, and so forth). In some embodiments, GUI 1310may also include a second display area 1312 for showing a summary of theplanogram compliance for all the products identified in the image. Theinformation displayed in GUI 1310 may help the non-employee individualto complete the task (e.g., rearranging the products placed on the storeshell, restocking certain products, etc,). Computing device 1201 mayfurther determine whether the task has been completed. For example,computing device 1201 may obtain one or more images of the store shelf(by, e.g., requesting the non-employee individual capture and upload animage of the store shelf) and determine whether the task is completedbased on the analysis of the one or more images. If computing device1201 that the task is completed, computing device 1201 may provide thereward to the non-employee individual. For example, computing device1201 may transmit a digital coupon corresponding to the incentive touser device 1301.

In some embodiments, the offer may be provided to two or morenon-employee individuals, and the individual who accepts the offer firstmay be offered the task. For example, computing device 1201 may providethe offer to two or more non-employee individuals and determine one ofthe non-employee individuals who accepts the offer first based on thetime of acceptance by the individual. Computing device 1201 may alsorescind the offer to other individual(s) by sending a notificationindicating the withdrawal of the offer to the user device associatedwith the individual(s) who did not get the offer.

In some embodiments, a non-employee individual among a plurality ofnon-employee individuals may be selected for receiving the offer tocomplete the task based on a property of the non-employee individuals(e.g., the height, physical strength, past behavior, level offamiliarity with the store, etc,, a property of the store shelf, type ofproducts placed on the store shelf, a property of the storage area forstoring one or more products associated with the store shelf, emergencylevel of the task, or the like, or a combination thereof. For example,computing device 1201 may select the at least one individual out of aplurality of alternative individuals using a property of the storeshelf. By way of example, fora task for rearranging the products on atop shelf, computing device 1201 may select a relatively tallnon-employee individual. As another example, computing device 1201 mayselect a more muscular individual for completing a task involving movinga heavy product.

In some embodiments, computing device 1201 may select a non-employeeindividual among a plurality of non-employee individuals to receive theoffer to complete the task based on one or more properties of the storeshelf. Exemplary properties of a store shelf may include informationrelated to at least one of products placed on the at least one storeshelf, empty spaces on the store shelf, a label attached to the storeshelf, a promotional sign associated with the store shelf, a height ofthe store shelf, a type of the store shelf, a location of the storeshelf within the retail store, a rate of customers passing in a vicinityof the store shelf, or the like, or a combination thereof. In someembodiments, any of the properties mentioned above may be used to selecta non-employee individual among a plurality of alternative non-employeeindividual. For example, users compatibility to the store shelf withdifferent properties may be obtained (for example, based on pastperformances, based on past training, based on past responsiveness,etc.), and the obtained compatibility to the store shelf may be used toselect the more compatible non-employee individuals from the pluralityof alternative non-employee individual as the non-employee individual toreceive the offer. In some embodiments, a machine learning model may betrained using training examples to select non-employee individuals basedon properties of shelves, and computing device 1201 may use the trainedmachine learning model to select the non-employee individual based onthe property of the store shelf. Alternatively or additionally,computing device 1201 may use an artificial neural network to anon-employee individual according to the property of the store shelf.

In some embodiments, the task may still be pending after a selected timeduration after the information related to the task and the incentive areoffered to the non-employee individual. For example, computing device1201 may receive at least one additional image captured after the offerwas provided to the at least one individual. By way of example,computing device 1201 may receive one or more images of the store shelfassociated with the task from a capturing device (as described elsewherein this disclosure) after the offer was provided to the individual.Computing device 1201 may also analyze the at least one additional imageto determine that the task is still pending (e.g., incomplete). In someembodiments, computing device 1201 may update the offer if the offer isdeclined by the non-employee individual (or no non-employee individualaccepts the offer). Computing device 1201 may update the incentive forthe task based on the determination that the at least one task is stillpending. For example, the incentive may include a discount, a coupon, anaward, a free product, or the like, or a combination thereof. Computingdevice 1201 may update the incentive based on a property of the storeshelf according to a type and/or level of the incentive. For example,computing device 1201 may escalate an incentive as time passes byupdating an incentive of $1 discount to an incentive of $2 discount. Insome embodiments, any of the properties of a store shelf disclosedelsewhere in this application may be used to determine the updatedincentive for the task. For example, the increase of the incentive maybe greater if the store shelf is located in a more remote area of theretail store. For example, computing device 1201 may update an incentiveof $1 discount to an incentive of $3 if the store shelf is more remote.In some embodiments, computing device 1201 may use a regression modelmay be used to calculate an updated amount associated with the updatedincentive for the task from the property of the store shelf. As anotherexample, computing device 1201 may update the original incentive for thetask (using the property of the store shelf as described above) byincreasing the original incentive based on the time passed since theinformation related to the task and the incentive was offered to the atleast one user, based on vicinity of users to the at least one storeshelf, or the like, or a combination thereof.

Alternatively or additionally, computing device 1201 may provide anadditional offer to at least one additional non-employee individualassociated with the retail store based on the determination that the atleast one task is still pending. For example, if computing device 1201determines that the task is still pending, computing device 1201 mayprovide the offer that is equal to the original offer to a secondnon-employee individual. Alternatively, computing device 1201 maydetermine an updated offer (as described elsewhere in this disclosure)and provide the updated offer to the second non-employee individual. Insome embodiments, if computing device 1201 determines that the task isstill pending, computing device 1201 may cancel the offer provided tothe non-employee individual. For example, computing device 1201 maydetermine that the task has not completed in a period of time and maycancel the offer by sending a notification to the user device associatedwith the non-employee individual who was rewarded the offer. In someembodiments, computing device 1201 may cancel the original offer afterreceiving an acceptance of the offer (an updated offer or an offerhaving the same terms as the original offer) rewarded to the secondperson.

In some embodiments, information relating to the non-employee individualmay be updated based on the performance (or non-performance) of thenon-employee individual. For example, computing device 1201 maydetermine that the task has been completed and update a record of thenon-employee individual.

In some embodiments, computing device 1201 may obtain an update to theproperty of the store shelf. For example, the update to the property ofthe store shelf may be read from memory, retrieved from a database,retrieved from a blockchain, received from an external device over acommunication network using a communication device, or the like, or acombination thereof. Alternatively or additionally, computing device1201 may determine an update to the property of the store shelf from ananalysis of image data of the store shelf captured after the informationrelated to the task and the incentive are offered to the non-employeeindividual. For example, the image data captured after the informationrelated to the task and the incentive is offered to the non-employeeindividual may be analyzed as described above to determine updatedproperty of the store shelf. In some embodiments, computing device 1201may also update the incentive and the offer for the task based on theupdated property of the store shelf. For example, computing device 1201may determine an updated incentive and offer based on the updatedproperty of the store shelf, as described elsewhere in this disclosure.In some embodiments, computing device 1201 may further transmit anupdated offer having the updated incentive to the non-employeeindividual as described elsewhere in this disclosure.

Embodiments of the present disclosure may provide a system foridentifying products in a retail store based on the image analysis ofone or more images depicting the products and automatically generatingan offer (e.g., a coupon) relating to the identified products. Forexample, the system may receive one or more images from an imagecapturing device attached to a store shelf, which may depict one or moreproducts. The system may analyze the images and determine a quantity ofthe products. The system may also automatically generate a coupon (e.g.,$1 off the price of one item of the products) based on the determinedquantity and transmit the coupon to a user device associated with one ormore customers. Although perishable products are used as exampleproducts to be identified, the systems and methods disclosed herein arenot limited to perishable products and may be implemented for othertypes of products, such as non-perishable products.

FIG. 15 illustrates an example system 1500 for generating offersrelating to identified products, consistent with the present disclosure.As illustrated in FIG. 15, system 1500 may include a computing device1501, database 1502, one or more user devices 1504 associated with oneor more customers 1503, one or more displays 1505, one or more shoppingcarts 1506, and a network 1507.

Computing device 1501 may be configured to obtain a set of imagesdepicting a store shelf having a plurality of perishable productsdisplayed thereon. Computing device 1501 may also be configured toanalyze the set of images and determine information about the displayedinventory of the perishable products based on the analysis of theimages. Computing device 1501 may further configured to obtain theinformation about the product type and determine at least one offerassociated with the product type using the information about thedisplayed inventory and the information about the product type.Computing device 1501 may also be configured to provide the offer to oneor more customers. In some embodiments, user device 1504 may beconfigured to receive one or more offers relating to a product type fromcomputing device 1501. For example, computing device 1501 may generate acoupon associated with a particular product and transmit an offer touser device 1504. Alternatively or additionally, computing device 1501may cause display 1505 and/or a display attached to cart 1506 to displaythe generated coupon.

Database 1502 may be configured to store information and/or data for oneor more components of system 1500. For example, database 1502 may beconfigured to store one or more images captured by capturing device 155.Computing device 1501 may access the one or more images stored indatabase 1502. As another example, database 1502 may store informationrelating to a retail store, information relating to one or more storeshelves of a retail store, information relating to one or morecustomers, information about one or more product types, or the like, ora combination thereof. Computing device 1501 may access informationstored in database 1502. For example, computing device 1501 may accessinformation about a particular product type (e,g., inventory informationof a particular product type).

Network 1507 may be configured to facilitate the communication betweenthe components of system 1500. For example, computing device 1501 maysend to and receive from user device 1504 information and/or data vianetwork 1507. Network 1507 may include a local area network (LAN), awide area network (WAN), portions of the Internet, an Intranet, acellular network, a short-ranged network (e.g., a Bluetooth™ basednetwork), or the like, or a combination thereof.

FIG. 16 is an example display showing an offer relating to a product,consistent with the present disclosure. As illustrated in FIG. 16, GUI1610 of user device 1504 may display a notification representing anoffer received from computing device 1501. In some embodiments, GUI 1610may also include the details of the offer (e.g., the discount, theprice, the location of the products associated with the offer, the timeperiod of the offer, or the like, or a combination thereof). Optionally,GUI 1610 may also include a map showing the location of the productsassociated with the offer or provide directions to the user to travel tothe location of the products associated with the offer (e.g., “proceedto end of aisle ten, then tum right,” etc.).

In some embodiments, user device 1504 may receive an updated offer fromcomputing device 1501, and GUI 1610 may display information relating tothe updated offer. For example, user device 1504 may receive anotification from computing device 1501 indicating that the previousoffer has expired, and GUI 1610 may display the information ofcancellation of the offer. As another example, user device 1504 mayreceive an updated offer including a coupon of a discount increased from$1 to $2, and GUI 1610 may display the information relating to theupdated offer.

FIG. 17 is a flowchart representing an exemplary method 1700 foridentifying perishable products in retail store 105 based on analysis ofimage data and for automatically generating offers relating to theidentified products in accordance with example embodiments of thepresent disclosure. For purposes of illustration, in the followingdescription, reference is made to certain components of system 1500 asdeployed in the configuration depicted in FIG. 15. For example, one ormore steps of method 1700 may be performed by computing device 1501. Itwill be appreciated, however, that other implementations are possibleand that other configurations may be utilized to implement the exemplarymethod. It will also be readily appreciated that the illustrated methodcan be altered to modify the order of steps, delete steps, or furtherinclude additional steps.

At step 1701, a set of images depicting a plurality of perishableproducts of at least one product type displayed on at least one shelvingunit in a retail store may be received. For example, computing device1501 may obtain a set of images depicting a store shelf having aplurality of perishable products displayed thereon. The one or moreimages may be captured by a capturing device as described elsewhere inthis disclosure. For example, the set of images may be acquired by aplurality of image sensors fixedly mounted in retail store 105. Detailsof the image capturing devices mounted in retail store 105 andconfigured to acquire the set of images are described with references toFIGS. 5A-5C. The set of images may be used to derive image data such asprocessed images (e.g., cropped images, video streams, and more) orinformation derived from the captured images (e.g., data about productsidentified in captured images, data that may be used to construct a 3Dimage, and more). In one embodiment, the processing device mayperiodically receive a set of images that corresponds with the currentinventory of the perishable products on the shelf. In this disclosure,the term “set of images” refers to one or more images. For example, theprocessing device may receive a set of images at predetermined timeintervals (e.g., every minute, every 5 minutes, every 15 minutes, every30 minutes, etc.). In another embodiment, the computing device mayreceive the set of images in response to a detection of a certainidentified event. In one example, the processing device may receive theset of images after an event, such as a detected lifting of a perishableproduct from the shelving unit. For example, computing device 1501 mayobtain the one or more images from a memory device, a database (e.g.,database 1502), an external device, or the like, or a combinationthereof, via a wired or wireless connection (e.g., network 1507).

At step 1703, the set of images may be analyzed to determine informationabout the displayed inventory of the plurality of perishable products.In some embodiments, computing device 1501 may analyze the set of imagesand determine information about the displayed inventory of theperishable products based on the analysis of the images. Exemplaryinformation about the displayed inventory of the perishable products mayinclude the information relating to products availability of at leastone type of the perishable products, the information relating to thequality of at least one of the perishable products, or the like, or acombination thereof.

The information relating to products availability of the perishableproducts may include an estimated current quantity of the perishableproducts, a predicted quantity of the perishable products at a selectedtime in the future, an estimated period during which the perishableproducts will not be nearly out of stock, or the like, or a combinationthereof. Alternatively or additionally, the information relating toproducts availability of the perishable products may include anindicator indicating the quantity of the perishable products. Forexample, computing device 1501 may determine a “few” indicatorindicating that there are less than five products based on the imageanalysis. As another example, computing device 1501 may determine a“moderate” indicator indicating that there are greater than five andless than 20 products, based on the image analysis. Alternatively oradditionally, the information relating to products availability of theperishable products may include an indicator indicating the demand ofthe perishable products.

The information relating to the quality of the perishable products mayinclude an indication that the perishable products are currently in agood condition, a predicted quality condition of the perishable productsat a selected time in the future (e.g., good, average, or bad), apredicted period during which the perishable products will remain in aquality condition, or the like, or a combination thereof. Alternativelyor additionally, the information relating to the quality of theperishable products may include an indicator indicating the quality ofat least one of the perishable products. For example, computing device1501 may determine a “good” indicator indicating that a first perishableproduct has a good (or fresh) condition and determine an “average”indicator indicating that a second perishable product has a not-so-goodcondition, based on the image analysis.

In some embodiments, one or more models may be used to determineinformation about the displayed inventory of the perishable products.For example, computing device 1501 may obtain a trained machine learningmodel that was trained using training examples to determine productsavailability data from images and/or videos. An example of such trainingexample may include an image and/or a video of perishable products,together with a label indicating the desired information to bedetermined about the perishable products. Computing device 1501 maydetermine the products availability data of at least one type of theperishable products using the trained machine learning model based onthe set of images. Alternatively or additionally, computing device 1501may obtain an artificial neural network configured to determine productsavailability data from images and/or videos, and determine the productsavailability data of at least one type of the perishable products usingthe artificial neural network based on the set of images. Alternativelyor additionally, computing device 1501 may obtain a trained machinelearning model that was trained using training examples to determineinformation relating to the quality of perishable products from imagesand/or videos. Computing device 1501 may determine the informationrelating to the quality of at least one of the perishable products usingthe trained machine learning model based on the set of images.Alternatively or additionally, computing device 1501 may obtain anartificial neural network configured to determine the quality ofperishable products from images and/or videos, and determine theinformation relating to the quality of at least one of the perishableproducts using the artificial neural network based on the set of images.

In some embodiments, information about the displayed inventory of theperishable products may be derived from data received from othernon-imaging sensors. For example, a weight sensor, which may beinstalled under (or integrated into) the store shelf, configured todetermine the weight of the perishable products. Computing device 1501may receive the weight data from the weight sensor and determine theinformation about the displayed inventory of the perishable productsbased on the weight data (e.g., the quantity of at least one type of theperishable products).

At step 1705, the information about the product type may be obtained ordetermined. For example, computing device 1501 may obtain theinformation about the product type from a memory device, a database(e.g., database 1502), an external device, or the like, or a combinationthereof, via a wired or wireless connection (e.g., network 1507).Alternatively or additionally, the processing device may analyze the setof images to determine one or more indicators associated with one ormore types of perishable products in retail store 105. The one or moretypes may include, for example, more than 25 types of perishableproducts, more than 50 types of perishable products, more than 100 typesof perishable products, etc. In one embodiment, the processing devicemay determine the information about the displayed inventory of theplurality of perishable products using a combination of image dataderived from the received set of images and/or data from one or moreadditional sensors configured to detect properties of perishableproducts placed on a store shelf. For example, the one or moreadditional sensors may include weight sensors, pressure sensors, touchsensors, light sensors, odors sensors, and detection elements asdescribed in relation to FIGS. 8A, 8B and 9, and so forth. Such sensorsmay be used alone or in combination with image capturing devices 125.For example, the processing device may analyze the data received fromodors sensors alone or in combination with images captured from theretail store, to estimate the condition or the ripeness level ofperishable products placed on a store shelving unit.

In some embodiments, analyzing the set of images to determineinformation about a displayed inventory of the plurality of perishableproducts may further comprise preprocessing the image data. In someexamples, the image data may be preprocessed by transforming the imagedata using a transformation function to obtain a transformed image data,and the preprocessed image data may include the transformed image data.For example, the transformation function may comprise convolutions,visual filters (such as low-pass filters, high-pass filters, band-passfilters, all-pass filters, etc.), nonlinear functions, and so forth. Insome examples, the image data may be preprocessed by smoothing the imagedata, for example, by using Gaussian convolution, using a median filter,and so forth. In some examples, the image data may be preprocessed toobtain a different representation of the image data. For example, thepreprocessed image data may include: a representation of at least partof the image data in a frequency domain; a Discrete Fourier Transform ofat least part of the image data; a Discrete Wavelet Transform of atleast part of the image data; a time/frequency representation of atleast part of the image data; a representation of at least part of theimage data in a lower dimension; a lossy compression representation ofat least part of the image data; a lossless compression representationof at least part of the image data; a time order series of any of theabove; any combination of the above; and so forth. In some examples, theimage data may be preprocessed to extract, edges, and the preprocessedimage data may include information based on and/or related to theextracted edges. In some examples, the image data may be preprocessed toextract visual features from the image data. Some examples of suchvisual features include edges, corners, blobs, ridges. Scale InvariantFeature Transform (SIFT) features, temporal features, and so forth. Oneof ordinary skill in the art will recognize that the image data may bepreprocessed using other kinds of preprocessing methods.

The information about the product type may include the information aboutthe product type, information about the inventory of the products of theproduct type, information about the costs associated with the producttype, information about previous sales of the product type, informationabout one or more related products or one or more related product types,or the like, or a combination thereof. For example, computing deviceI501 may obtain information about the product type, including, forexample, the shelf life of the product type.

The information about the inventory of the products of the product typemay include the current inventory at the retail store, predictedinventory at the retail store at a selected time in the future, thecurrent inventory at another retail store, predicted inventory atanother retail store at a selected time in the future. For example,computing device 1501 may obtain information about additional perishableproducts from the product type available in a storage area of the retailstore. As another example, computing device 1501 may obtain theinformation about additional perishable products from the product typescheduled to be delivered to the retail store. By way of example,computing device 1501 may obtain or determine an estimated arrival timeof the additional perishable products. Alternatively or additionally,computing device 1501 may obtain or determine an estimated time toproduce additional perishable products (e.g., baked products).

The information about the costs associated with the product type mayinclude a current price of the product type, an estimated cost if aportion or all of the perishable products are discarded, or the like, ora combination thereof.

The information about one or more previous sales of the product type mayinclude the information of at least one previous sale of products of theproduct type, including, for example, the time of the sale (e.g.,minute, hour, day, month, etc.), the discount of the sale, the on-saleprice of the sale, the time period of the sale, the quantity sold duringthe sale, or the like, or a combination thereof. Alternatively oradditionally, the information about one or more previous sales of theproduct type may include the information relating to the demographiccharacteristic of customers that bought, including, for example, age,age group, gender, residence, or the like, or a combination thereof.

The information about, one or more related products or one or morerelated product types may include the information of one or more otherproduct types that a customer is also likely to buy if he or she buysthe product type. For example, computing device 1501 may obtain (ordetermine) the information about nachos (i.e., the product type) andalso the information about a related product type (e.g., avocados). Insome embodiments, the information about one or more related products orone or more related product types may be determined based on cashierdata of past sales associated with one or more customers. Alternativelyor additionally, the information about one or more related products orone or more related product types may be determined based on the imageanalysis of one or more images depicting the products found in a cart ofone or more customers. For example, one or more images may be capturedby a built-in camera of a cart, a camera attached to a card, or astationary camera, or the like, or a combination thereof. Computingdevice 1501 may analyze the one or more images and determine theinformation based on the image analysis.

Alternatively or additionally, the processing device may obtaininformation about additional perishable products scheduled to bedisplayed on the at least one shelving unit. Consistent with the presentdisclosure, the processing device may analyze scheduling data todetermine one or more indicators associated with one or more types ofperishable products scheduled to be delivered to retail store 105. Theinformation about additional perishable products may include at leastone of: a schedule of arrivals of the additional products (e.g., dates,times, updates from a supplier, etc.), known orders of the additionalproducts (e.g., quantity of scheduled products), delivery records (e.g.,the reports on the delivery progress of the additional products), costdata (e.g., the different costs associated with the additionalproducts), calendar data (e.g., holidays, sport games, etc.), historicalproduct turnover data (e.g., the amount of avocados sold per lastApril), average turnover (e.g., per hour in a day, per day in a week,per special events), and more. In one embodiment, the information aboutthe additional perishable products may be obtained from multiple sources(e.g., market research entity 110, supplier 115, and multiple retailstores 105) and may include different types of parameters. In someembodiments, the obtained information about the additional perishableproducts may be stored digitally in memory device 226, stored indatabase 140, received using network interface 206 through communicationnetwork 150, received from a user, determined by processing device 202,and so forth.

At step 1707, the information about the displayed inventory and theinformation about the product type may be used to determine at least oneoffer associated with the product type. For example, computing device1501 may determine at least one of associated with the product typeusing the information about the displayed inventory (described elsewherein this disclosure) and the information about the product type(described elsewhere in this disclosure). In some embodiments, computingdevice 1501 may determine the tents of an offer, including, for example,one or more products associated with the offer, a discount associatedwith the offer, a price of the product type(s), a time period of theoffer, one or more customers who are to receive the offer, the quantityof the products associated with the offer, one or more restrictionsassociated with the offer (e.g., the quantity of the products that onecustomer can purchase), or the like, or a combination thereof.

For example, the information about the displayed inventory may includequality indicators for the plurality of perishable products. Computingdevice 1501 may determine the quality indicators for the plurality ofperishable products based on an analysis of the set of images and usethe determined quality indicators for the plurality of perishableproducts to determine the offer associated with the product type (e.g.,a sale for croissants). Optionally, computing device 1501 may also usethe determined quality indicators to determine a shelf-lift estimationof the plurality of perishable products, and to determine the offerassociated with the product type based on shelf-life estimation of theplurality of perishable products. For example, computing device 1501 maydetermine that a shelf-life estimation of croissants is 45 minutes anddetermine a sale of croissants starting in 10 minutes. Alternatively oradditionally, computing device 1501 may obtain information aboutadditional perishable products from the product type available in astorage area of the retail store. Computing device 1501 may also use theinformation about the displayed inventory and the information about theadditional perishable products available in the storage area todetermine the offer associated with the product type. By way of example,computing device 1501 may determine that there is a large quantity ofavocados in the storage based on the obtained information and determinea sale of avocados at a discount. Alternatively or additionally,computing device 1501 may obtain the information about additionalperishable products from the product type scheduled to be delivered tothe retail store and use the information about the displayed inventoryand the information about the additional perishable products scheduledto be delivered to the retail store to determine the offer associatedwith the product type. Alternatively or additionally, computing device1501 may obtain information about costs associated with the product typeand use the information about the displayed inventory and the costsassociated with the product type to determine the offer associated withthe product type. Alternatively or additionally, computing device 1501may obtain a current price for the product type and use the informationabout the displayed inventory and the current price for the product todetermine the over associated with the product type. Alternatively oradditionally, computing device 1501 may obtain the details of at leastone previous sale of products from the product type and use theinformation about the displayed inventory and the details of theprevious sale to determine the offer associated with the product type.By way of example, computing device 1501 may obtain the details of atleast one previous sale including at least one demographiccharacteristic or customers that bought the product type. Computingdevice 1501 may determine an offer to at least one customer of theretail store based on a predetermined demographic characteristicassociated with the customer (e g., sending a coupon of pizza to youngpeople, but not to older people).

In some embodiments, the determination of an offer may be further basedon the information relating to a particular customer. For example,computing device 1501 may obtain (or determine) information indicativeof products from multiple product types that a customer is likely to buy(described elsewhere in this disclosure), and use the information aboutthe displayed inventory, the stored information about the product type,and the information indicative of products that the customer is likelyto buy to determine a personalized offer for the customer. In someembodiments, computing device 1501 may obtain the information relatingto a particular customer from a memory device, a database (e.g.,database 1502), an external device, or the like, or a combinationthereof, via a wired or wireless connection (e.g., network 1507).

In some embodiments, the information relating to a particular customerused for generating an offer to a customer may include a personalizedprofile of the customer. The personalized profile of the customer mayinclude information based on past purchases of the customer, and theinformation based on past purchases of the customer and/or theimage-based products availability data may be used to generate an offerto the customer (e.g., a coupon). Alternatively or additionally, thepersonalized profile of the customer may include levels of associationof the customer to a plurality of retail stores (e.g., based on pastpurchases, based on past geo-location information related to the user,etc). In some embodiments, the image-based products availability datamay include image-based products availability records related to theplurality of retail stores, and the levels of association of thecustomer to the plurality of retail stores and/or the image-basedproducts availability records related to the plurality of retail storesmay be used to generate an offer. Alternatively or additionally, thepersonalized profile of the customer may include an indication that thecustomer is near a retail store. In some embodiments, the image-basedproducts availability data may include image-based products availabilityrecords related to the retail store, and the indication that thecustomer is near the retail store and/or the image-based productsavailability records related to the retail store may be used to generatean offer. Alternatively or additionally, the personalized profile of thecustomer may include a current location of the customer within a retailstore. In some embodiments, the image-based products availability datamay include image-based products availability record related to theretail store, and the current location of the customer within the retailstore and/or the image-based products availability record related to theretail store may be used to generate an offer. Alternatively oradditionally, the personalized profile of the customer may includeinformation related to times of shopping events related to the customer.The information related to times of shopping events related to thecustomer may be used to predict a time of a future shopping event of thecustomer. In some embodiments, the image-based products availabilitydata may be used to predict future products availability data related tothe predicted time of the future shopping event of the customer. Thepredicted future products availability data may be used to generate anoffer.

In some embodiments, computing device 1501 may be configured to identifya specific customer that will likely to be interested in discountedproducts from the product type. For example, computing device 1501 mayidentify the specific customer based on a personalized profile of thecustomer may be accessed (described elsewhere in this disclosure) andother information. Computing device 1501 may also determine an offer forthis customer based on the information about the displayed inventory andthe information about the product type may be used to determine at leastone offer associated with the product type as described elsewhere inthis disclosure.

At step 1709, the over may be provided to one or more individuals. Forexample, computing device 1501 may obtain the information relating tothe offer and transmit the offer to user device 1504 associated withcustomer 1503.

In some embodiments, information related to an offer (e.g., a generatedcoupon) may be stored in memory or a database (e.g., database 1502).Alternatively or additionally, a generated offer may be transmittedthrough a communication network using a communication device to, forexample, an external device, a user device associated with one or morecustomers, a computer associated with the retail store, one or moredisplays in the retail store, a computer associated with one or moreother retail stores, one or more displays associated with one or moreother retail stores, or the like, or a combination thereof. For example,a generated coupon may be provided to the customer, for example, using amobile device associated with the customer (such as a cell phone, smartglasses, tablet, display on a cart associated with the customer, etc.),through a dedicated application, through a website, through an email,through mail, or the like, or a combination thereof. By way of example,computing device 1501 may transmit a message to a mobile communicationdevice associated with the customer to display information related tothe offer (e.g., a notification illustrated in GUI 1610 shown in FIG.16). Alternatively or additionally, computing device 1501 may cause anaugmented reality system associated with the customer to displayinformation related to the offer. Alternatively or additionally,computing device 1501 may transmit information relating to the offer toone or more displays in the retail store for displaying the offer. Byway of example, computing device 1501 may cause a display attached to acart associated with the customer to display information related to theoffer. In some embodiments, the display may also display informationrelated to the generated offer.

In some embodiments, computing device 1501 may generate a customizedoffer for a particular customer as described elsewhere in thisdisclosure. Computing device 1501 may provide the customized offer tothe customer by, for example, transmitting the offer to a user deviceassociated with the customer.

In some embodiments, an offer may be modified or canceled. For example,computing device 1501 may receive one or more additional imagesdepicting the shelving unit in the retail store after the offer wasprovided to at least one customer. Computing device 1501 may alsoanalyze the one or more additional images to determine current inventoryof products from the product type as described elsewhere in thisdisclosure. When the current inventory of products from the product typeis less than a threshold, computing device 1501 may modify or cancel theoffer. For example, computing device 1501 may inform one or morecustomers that the offer is expired. Alternatively or additionally,computing device 1501 may use the current inventory of products of theproduct type and the stored information to determine at least oneupdated offer associated with the product type. Computing device 1501may update the offer by modifying one or more products specified in theoffer, a discount, a price, a quantity that a customer can buy, the timeperiod of the offer, or the like, or a combination thereof. Computingdevice 1501 may also transmit the updated offer to an external device, auser device associated with one or more customers, a computer associatedwith the retail store, one or more displays in the retail store, acomputer associated with one or more other retail stores, one or moredisplays associated with one or more other retail stores, or the like,or a combination thereof.

Embodiments of the present disclosure may provide a system fordynamically identifying a deviation of the actual placement of productson a store shelf from a current planogram based on the analysis ofimages. The system may be configured to dynamically select, among aplurality of planograms, a planogram for determining whether theproducts placed on the store shelf conform to the identified planogram.For example, a planogram may be used for determining the compliance ofthe produced placed on the store shelf during a certain period of timeof a day (e.g., 8:00 to 11:00 a.m.). The system may receive a set ofimages depicting products on the store shelf and select the planogrambased on the time when the first set of images are captured. The systemmay also be configured to analyze the images and determine whether theproducts on the store shelf conform with the planogram.

FIG. 18 is a schematic illustration of an example system for identifyingchanges in planograms, consistent with the present disclosure. FIG. 18illustrates an example system 1800 for generating offers relating toidentified products, consistent with the present disclosure. Asillustrated in FIG. 18, system 1800 may include a computing device 1801,database 1802, one or more user devices 1804 associated with one or moreemployees 1803 of a retail store, and a network 1805.

Computing device 1801 may be configured to obtain a first planogram andreceive a first set of images depicting a first plurality of products ona shelving unit (or a store shelf). Computing device 1801 may also beconfigured to analyze the first set of images to determine an actualplacement of at least portion of the first plurality of productsdisplayed on the shelves of the retail store at the first time.Computing device 1801 may further be configured to identify a deviationof an actual placement of at least sonic of the first plurality ofproducts from the desired placement of products associated with thefirst planogram, based on the first planogram and the determined actualplacement of at least some of the first plurality of products displayedon the store shelf. Computing device 1801 may also be configured togenerate a first user-notification associated with the deviation of theactual placement of the at least some of the first plurality of productsfrom the desired placement of products associated with the firstplanogram computing device 1801 may further be configured to receive asecond set of images captured at a second time and depicting a secondplurality of products displayed on the at least one of the shelves ofthe retail store. Computing device 1801 may also be configured toanalyze the second set of images to determine an actual placement of thesecond plurality of products displayed on the shelves of the retailstore at the second time. Computing device 1801 may further beconfigured to access a rule specifying which planogram (e.g., the firstplanogram, the second planogram, or anther planogram) is to be used fordetermining whether an arrangement associated with the second pluralityof products are in planogram compliance. Computing device 1801 may alsobe configured to analyze the second set of images to determine whetheran arrangement associated with the second plurality of products conformsto the particular planogram (e,g., the second planogram) rather than thefirst planogram may be determined. If computing device 1801 determinesthat an arrangement associated with the second plurality of productsconforms to the second planogram, computing device 1801 may withhold theissuance of a second notification indicating the deviation associatedwith the first planogram.

User device 1804 may be configured to receive information from computingdevice 1801 via, for example, network 1805. For example, user device1804 may be configured to receive a notification associated with adeviation of the actual placement of products on a store shelf fromcomputing device 1801. In some embodiments, user device 1804 may includean output device to show the notification received from computing device1801. For example, user device 1804 may include a display configured toprovide a GUT showing the notification associated with the deviation.

Database 1802 may be configured to store information and/or data for oneor more components of system 1800. For example, database 1802 may beconfigured to store one or more images captured by capturing device 185(not shown) as described elsewhere in this disclosure. Computing device1801 may access the one or more images stored in database 1802. Asanother example, database 1802 may store one or more planograms. Aplanogram may describe or specify a desired placement of products onshelves of a retail store during a time period, as described elsewherein this disclosure.

Network 1805 may be configured to facilitate the communication betweenthe components of system 1800. For example, computing device 1801 maysend to and receive from user device 1804 information and/or data vianetwork 1805. Network 1805 may include a local area network (LAN), awide area network (WAN), portions of the Internet, an Intranet, acellular network, a short-ranged network (e.g., a Bluetooth™ basednetwork), or the like, or a combination thereof.

FIG. 19 is an example output for displaying a deviation of an actualplacement of products displayed on a store shelf from the desiredplacement of products associated with a planogram. For example,computing device 1801 may be configured to transmit a notificationassociated with a deviation to user device 1804 associated with employee1803 illustrated in FIG. 18. An output device of user device 1804 (e.g,,a display) may display a GUI 1910 showing the notification associatedwith the deviation, as illustrated in FIG. 19. In some embodiments, GUI1910 may include a first display area 1911 for showing a display of oneor more images depicting a store shelf with augmented markings indictingthe deviation (e.g., correct place, misplaced, not in planogram, empty,and so forth). GUI 1910 may also include a second display area 1912 forshowing a summary of the deviation.

FIGS. 20A and 20B show a flowchart of an exemplary method 2000 forissuing a notification of planogram deviation, consistent with thepresent disclosure. At step 2001, a first planogram may be obtained. Forexample, computing device 1801 may obtain a first planogram from amemory device, a database (e.g., database 1502), an external device, orthe like, or a combination thereof, via a wired or wireless connection(e,g., network 1507). A planogram may describe a desired placement ofproducts on shelves of a retail store during a time period, as describedelsewhere in this disclosure, in some embodiments, computing device 1801may be configured to access a plurality of planograms stored in adatabase (e.g., database 1802). The planograms may include a firstplanogram and a second planogram. In some embodiments, the planogramsaccessed by computing device 1801 may include three or more planograms.In some embodiments, the plurality of planograms may be indicative ofthe desired placement of products on shelves of the retail store atdifferent times of a day. For example, the first planogram may beassociated with a first time period and the second planogram may beassociated with a second time period subsequent to the first timeperiod. By way of example, the First planogram may be associated with afirst time period between 08:00 to 16:00, and the second planogram maybe associated with a second time period between 16:00 to 22:00. In someembodiments, one or more planograms obtained by computing device 1801may be indicative of the desired placement of products on shelves of theretail store at different times (or time periods) of a day, differentdays of a week, different weeks of a month, different months of a year,different years, or the like, or a combination thereof. Alternatively oradditionally, at least one of the planograms may be received fromanother retail store. For example, computing device 1801 may receive aplanogram from a computing device associated with another retail store.The planogram from another retail store may be more updated or recent.Alto natively or additionally, computing device 1801 may receive one ormore images depicting one or more shelves in another retail store anddetermine a planogram (as one of the planograms described above) basedon the analysis of the one or more images received. For example,computing device 1801 may determine an actual arrangement of theproducts on the one or more shelves in the retail store based on theimage analysis, determine a planogram based on the actual arrangement ofthe products. In some embodiments, computing device 1801 may alsodetermine a time period associated with the determined planogram.

At step 2002, a first set of images captured at a first time anddepicting a first plurality of products displayed on at least one of theshelves of the retail store may be received. For example, a first set ofimages depicting a first plurality of products displayed on at least oneof the shelves of the retail store may be captured by a capturing device(e.g., capturing device 155) at a first time. Computing device 1801 mayreceive the first set of images from a capturing device, a memorydevice, a database (e,g., database 1502), an external device, or thelike, or a combination thereof; via a wired or wireless connection(e,g,, network 1507). By way of example, computing device 1801 mayinclude a data interface configured to receive the first set (and: orother set) of images from one or more image sensors fixedly mounted onthe at least one retail shelving unit.

At step 2003, the first set of images may be analyzed to determine anactual placement of a first plurality of products. For example,computing device 1801 may be configured to analyze the first set ofimages to determine an actual placement of at least a portion of thefirst plurality of products displayed on the shelves of the retail storeat the first time, as described elsewhere in this disclosure.

In some embodiments, one or more computer models or algorithms may beused to determine the actual placement of products on a store shelf. Forexample, computing device 1801 may obtain a trained machine learningmodel that was trained using training examples to determine the actualplacement of products on a store shelf from images and/or videos.Computing device 1801 may determine the actual placement of the firstplurality of products on the store shelf using the trained machinelearning model based on the first set of images. Alternatively oradditionally, computing device 1501 may obtain an artificial neuralnetwork configured to determine the actual placement of products on astore shelf from images and: or videos, and determine the actualplacement of products on a store shelf using the artificial neuralnetwork based on the first set of images. Alternatively or additionally,computing device 1801 may use one or more computer vision algorithms toanalyze the first set of images to determine the actual placement of thefirst plurality of products on the store shelf.

At step 2004, a first deviation of an actual placement of at least someof the first plurality of products may be determined. For example,computing device 1801 may be configured to identify a deviation of anactual placement of at least some of the first plurality of productsfrom the desired placement of products associated with the firstplanogram, based on the first planogram and the determined actualplacement of at least some of the first plurality of products displayedon the store shelf. By way of example, computing device 1801 may detectone or more deviations based on differences between the first planogramand the actual placement of the at least some of the plurality ofproducts. In some embodiments, a deviation may relate to one or morecharacteristics of planogram compliance, which are described elsewherein this disclosure. For example, a deviation may include product facing,product placement, planogram compatibility, price correlation, promotionexecution, product homogeneity, restocking rate, and planogramcompliance of adjacent products, or the like, or a combination thereof.

In some embodiments, if computing device 1801 does not identify anydeviation, computing device 1801 may not issue any notification (e.g., anotification associated with a deviation as described elsewhere in thisdisclosure). Alternatively, computing device 1801 may issue anotification indicating that no deviation associated with the storeshelf is found. For example, computing device 1801 may transmit anotification indicating that no deviation associated with the storeshelf is found to a device associated with an employee of the retailstore, a device associated with the retail store, or the like, or acombination thereof. Alternatively or additionally, computing device1801 may store the information indicating no deviation was found at thefirst time.

At step 2005, a first notification associated with the first deviationmay be issued. For example, computing device 1801 may be configured togenerate a first user-notification associated with the deviation of theactual placement of the at least some of the first plurality of productsfrom the desired placement of products associated with the firstplanogram. Computing device 1801 may also be configured to transmit thefirst user-notification to a handheld device associated with anemployee. In some embodiments, the notification may also include detailsof the deviation of the actual placement of the at least some of thefirst plurality of products from the desired placement of productsassociated with the first planogram. By way of example, computing device1801 may transmit a notification associated with the first deviation touser device 1804 associated with employee 1803 illustrated in FIG. 18.An output device of user device 1804 (e.g., a display) may display a GUI1910 showing a first display area 1911 for showing a display of one ormore images depicting a store shelf with augmented markings indictingthe deviation (e.g., correct place, misplaced, not in planogram, empty,and so forth). GUI 1910 may also include a second display area 1912 forshowing a summary of the deviation. Alternatively or additionally,computing device 1801 may be configured to transmit the firstnotification to a computer associated with the retail store, which mayinclude an output device configured to display the notification.

At step 2006, after the first user-notification is issued, a second setof images captured at a second time and depicting a second plurality ofproducts displayed on the at least one of the shelves of the retailstore may be received. For example, a second set of images may becaptured by a capturing device (e.g., capturing device 155) at a secondtime, which may depict a second plurality of products displayed on theat least one of the shelves of the retail store. Computing device 1801may receive the second set of images from the capturing device, a memorydevice, a database (e.g., database 1502), an external device, or thelike, or a combination thereof, via a wired or wireless connection(e.g., network 1507). In some embodiments, the second plurality ofproducts may be the first plurality of products depicted in the firstset of images. Alternatively, the second plurality of products may bedifferent from the first plurality of products depicted in the first setof images.

In some embodiments, as described elsewhere in this disclosure, thefirst planogram may be associated with a first time period and thesecond planogram may be associated with a second time period subsequentto the first time period. The second time at which the second set ofimages are captured may be included in the first time period. By way ofexample, the first planogram may be associated with a first time periodbetween 08:00 to 16:00, and the second planogram may be associated witha second time period between 16:00 to 22:00. The store employee maychange the arrangement of the products according to the second planogramat 15:00. The capturing device may be configured to capture the secondset of images at 15:05 (i.e., the second time).

In some embodiments, computing device 1801 may include a data interfaceconfigured to receive the second set (and/or the first set) of imagesfrom one or more image sensors fixedly mounted on the at least oneretail shelving unit.

At step 2007, a second deviation of the actual placement of at leastsome of the second plurality of products from the desired placement ofproducts associated with the first planogram may he determined. Forexample, computing device 1801 may be configured to analyze the secondset of images to determine an actual placement of the second pluralityof products displayed on the shelves of the retail store at the secondtime. Computing device 1801 may also be configured to identify adeviation of the actual placement of at least some of the secondplurality of products from the desired placement of products associatedwith the first planogram, as described elsewhere in this disclosure(e.g., the description in connection with steps 2003 and 2004). By wayof example, the first planogram may indicate that a group of firstproducts are supposed to be placed on a shelving unit. Computing device1801 may determine that a group of second products are placed on theshelving unit where a group of first products were supposed to beplaced, and identify this deviation as a deviation of the actualplacement of at least some of the second plurality of products from thedesired placement of products associated with the first planogram. Insome embodiments, a machine learning model may be trained using trainingexamples to determine deviations of actual placements of products fromplanograms from images and/or videos, and from planograms. An example ofsuch training example may include a planogram, and an image and/or avideo depicting products on one or more retail shelves, together a labelindicating a deviation of the placement of the products in the imageand/or in the video from the planogram. Additionally or alternatively tostep 2003 and/or step 2004, method 2000 may use the trained machinelearning model to analyze the first set of images received by step 2002and the first planogram received by step 2001 to determine the firstdeviation. Additionally or alternatively to step 2007, method 2000 mayuse the trained machine learning model to analyze the second set ofimages received by step 2006 and the first planogram received by step20W to determine the second deviation.

In some embodiments, computing device 1801 may be configured todetermine the second time is during a transition period between thefirst planogram and a different planogram (e.g., the second planogram ora third planogram). In response to the determination that the secondtime is during the transition period, method 2000 may proceed to step2011, at which computing device 1801 may withhold issuance of the seconduser-notification associated with the deviation of the actual placementof the at least some of the second plurality of products from thedesired placement of products associated with the first planogram.

At step 2008, a rule for determining whether the at least some of thesecond plurality of products are in planogram compliance may beaccessed. For example, computing device 1801 may be configured to accessa rule specifying which planogram (e.g., the first planogram, the secondplanogram, the third planogram, or anther planogram) is to be used fordetermining whether an arrangement associated with the second pluralityof products are in planogram compliance. By way of example, a rule mayspecify that, instead of the first planogram, the second planogramshould be used for determining whether the second plurality of productsare in planogram compliance. Computing device 1801 may be configured touse the second planogram to determine whether an arrangement associatedwith the second plurality of products conforms to the second planogramrather than to the first planogram. Alternatively, if the rule specifiesthat the first planogram should be used, method 2000 may proceed to step2012 (the dashed line coming out of box 2008 and pointing to box 2012),and a second notification associated with the second deviationidentified at step 2007 may be issued.

In some embodiments, computing device 1801 may receive the rule from amemory device, a database (e.g., database 1802), an external device, orthe like, or a combination thereof, via a wired or wireless connection(e.g., network 1805).

In some embodiments, the rule may specify a particular planogram to usefor determining whether at least some of the second plurality ofproducts are in planogram compliance, based on the time of the secondset of images being captured (i.e., the second time), the time elapsedsince the first plurality of products were last arranged according tothe first planogram, the data from another retail store indicative of acurrent planogram being used in the another retail store, or the like,or a combination thereof. Computing device 1801 may access the rule andselect a particular planogram (e.g., among a group of possible ofplanograms) according to the rule. Computing device 1801 may alsodetermine whether at least sonic of the second plurality of products arein planogram compliance according to the selected planogram, asdescribed elsewhere in this disclosure. For example, the rule mayspecify the second planogram to use when it is the Christmas season, andcomputing device 1801 may select the second planogram to use fordetermining whether at least some of the second plurality of productsare in planogram compliance if the second time is during the Christmasseason or in December.

In some embodiments, the rule may be used to analyze the image data(e.g., the first set of images, the second set of images, or both) todetermine whether to switch from the first planogram to the secondplanogram. For example, the image data may be analyzed (for exampleusing a machine learning model trained using training examples, using anartificial neural network, using computer vision algorithms, etc.) todetermine whether a trigger specified by the rule is detected and/or acondition specified by the rule is met. Alternatively or additionally,the image data may comprise a depiction of the specific area of thestore shelf, and the rule may be based, at least in part, on a level ofplanogram compliance related to the specific area of the store shelfdetermined by analyzing the image data. Alternatively or additionally,the image data may comprise a depiction of the specific area of thestore shelf, and the rule may be based, at least in part, onavailability of products on the specific area of the store shelfdetermined by analyzing the image data. Alternatively or additionally,the image data may comprise a depiction of the specific area of theshelf, and the rule may be based, at least in part, on a flow ofproducts corresponding to the specific area of the store shelfdetermined by analyzing the image data. Alternatively or additionally,the image data may comprise a depiction of customers entering the retailstore, and the rule may be based, at least in part, on at least oneproperty of the customers entering the retail store determined byanalyzing the image data.

In some embodiments, when the rule for switching one planogram toanother planogram is activated, computing device 1801 may transmit anotification indicating the switch of the planograms to, for example, adevice associated with one or more employees, a device associated withthe retail store, a device associated with a non-employee (e.g., acustomer) or the like, or a combination thereof. For example, computingdevice 1801 may transmit a notification to user device 1804 indicatingthat the second planogram is in effect. In some embodiments, thenotification may include at least one change to the products placed onthe store shelf (e.g., at least one product is needed to be replaced ona specific area of the store shelf). Alternatively or additionally, thenotification may include an instruction to move the store shelf (e.g.,to be moved by automatically using a mechanical mechanism, by a storeemployee, by a non-employee, or the like, or a combination thereof).

In some embodiments, computing device 1801 may be configured todetermine a likelihood of switching from the first planogram to anotherplanogram (e.g., the second planogram). If computing device 1801determines that the likelihood is smaller than threshold, computingdevice 1801 may issue a user-notification associated with a deviation ofthe actual placement of the at least some of the second plurality ofproducts from the desired placement of products associated with thefirst planogram. On the other hand, if the likelihood is greater thanthe threshold, computing device 1801 may use the second planogram fordetermining whether the products conform to the second planogram(instead of the first planogram), as described elsewhere in thisdisclosure.

At step 2009, a particular planogram (e.g., the second planogram) may beobtained based on the rule. For example, computing device 1801 mayobtain the second planogram based on the rule. Computing device 1801 mayreceive the second planogram from a memory device, a database (e.g.,database 1802), an external device, or the like, or a combinationthereof, via a wired or wireless connection (e.g., network 1805).Alternatively or additionally, step 2008 may be excluded from method2000, and step 2009 may obtain the particular planogram (such as thesecond planogram, the third planogram, other planograms, and so forth),for example by reading the particular planogram from a memory device, bygenerating the particular planogram (for example, from a realogram,according to inputs from a user, by analyzing past image data, and soforth), from an external device (for example, through a communicationdevice), by selecting the particular planogram of a plurality ofalternative planograms, and so forth. In some embodiments, the firstplanogram may indicate of a group of first products are supposed to beplaced on the shelving unit, and the second planogram may indicate thata group of second products are supposed to be placed on the shelvingunit. Optionally, the group of second products may include more thanfive products from a same type. Alternatively or additionally, the groupof second products may include products from at least two types ofproducts.

At step 2010, whether an arrangement associated with the secondplurality of products conforms to the particular planogram (e.g., thesecond planogram) rather than the first planogram may be determined. Forexample, computing device 1801 may be configured to analyze the secondset of images to determine an actual placement of at least a portion ofthe second plurality of products displayed on the shelves of the retailstore, as described elsewhere in this disclosure. Computing device 1801may also be configured to determine whether an arrangement associatedwith the second plurality of products conforms to the second planogram,based on the actual placement of at least portion of the secondplurality of products and the second planogram (as described elsewherein this disclosure). For example, computing device 1801 may determine anarrangement associated with the second plurality of products conforms tothe second planogram when the percent of the group of second productsplaced on the at least one of the shelves of the retail store conformsto the second planogram equals or exceeds a threshold (e.g., 90%). Insome examples, a machine learning model may be trained using trainingexamples to select most conforming planograms to actual arrangements ofproducts of pluralities of alternative planograms. An example of suchtraining example may include a plurality of alternative planograms (suchas two planograms, three planograms, more than three planograms, etc.)and an indication of an actual arrangement of products, together with alabel indicating the planogram of the plurality of alternativeplanograms that the actual placement of products is most conformingwith, or together with a label indicating that the actual placement ofproducts does no conform with any of planogram of the plurality ofalternative planograms. Further, in one example, step 2010 may use thetrained machine learning model to analyze the arrangement associatedwith the second plurality of products and determine whether thearrangement conforms with the first planogram, with the secondplanogram, with the third planogram, with another planogram, with noplanogram, and so forth. In some examples, a machine learning model maybe trained using training examples to select most conforming planogramsto actual arrangements of products of pluralities of alternativeplanograms based on images and/or videos of the actual arrangements ofproducts. An example of such training may include a plurality ofalternative planograms (such as two planograms, three planograms, morethan three planograms, etc.) and an image and/or a video of an actualarrangement of products, together with a label indicating the planogramof the plurality of alternative planograms that the depicted actualplacement of products is most conforming with, or together with a labelindicating that the depicted actual placement of products does noconform with any of planogram of the plurality of alternativeplanograms. Further, in one example, step 2010 may use the trainedmachine learning model to analyze the second set of images received bystep 2006 and determine whether the arrangement conforms with the firstplanogram, with the second planogram, the third planogram, with anotherplanogram, with no planogram, and so forth. If computing device 1801determines that an arrangement associated with the second plurality ofproducts conforms to the second planogram, method 2000 may proceed tostep 2011. At step 2011, computing device 1801 may withhold the issuanceof a second notification indicating the deviation associated with thefirst planogram (identified at step 2007). On the other hand, ifcomputing device 1801 determines that an arrangement associated with thesecond plurality of products does not conform to the second planogram,method 2000 may proceed to step 2012.

At step 2012, computing device 1801 may issue a second notificationindicating a deviation associated with at least some products of thesecond plurality of products from the first planogram (e.g., a deviationidentified at step 2007). Alternatively or additionally, computingdevice 1801 may issue a second notification indicating a deviationassociated with at least some products of the second plurality ofproducts from the second planogram, which may be identified based on theanalysis of the second set of images as described elsewhere in thisdisclosure. For example, computing device 1801 may analyze the secondset of images and determine an actual placement of at least some of thesecond plurality of products from the desired placement of productsassociated with the second planogram based on the image analysis.Computing device 1801 may also identify a deviation of the actualplacement of at least some of the second plurality of products from thedesired placement of products associated with the second planogram. Byway of example, computing device 1801 may detect one or more deviationsbased on differences between the second planogram and the actualplacement of the at least some of the second plurality of products.

In some embodiments, computing device 1801 may access a plurality ofplanograms (as described elsewhere in this disclosure), which mayinclude at least three planograms (e.g., the first planogram, the secondplanogram, and a third planogram). Computing device 1801 may analyze thesecond set of images and the at least three planograms to determinewhich planogram was used for arranging the second plurality of productsat a time between the first time and the second time, for example usinga machine learning model as described above. In response to adetermination that the second plurality of products were arrangedaccording to the second planogram and that there is a deviation of theactual placement of the second plurality of products from the desiredplacement of products associated with the second planogram, computingdevice 1801 may issue a user-notification associated with the secondplanogram. Alternatively, in response to a determination that the secondplurality of products were arranged according to another planogram andthat there is a deviation of the actual placement of the secondplurality of products from the desired placement of products associatedwith the another planogram, computing device 1801 may issue auser-notification associated with the another planogram.

In some embodiments, a third set of images may be captured at a thirdtime after the determination of whether the arrangement associated withthe second plurality of products conforms to the second planogram oranother planogram (e.g., a third planogram). The images may be capturedby a capturing device as described elsewhere in this disclosure.Computing device 1801 may be configured to receive the third set ofimages and analyze the third set of images to determine the actual placeof products on the shelves of the retail store at the second time, asdescribed elsewhere in this disclosure. Computing device 1801 may alsobe configured to determine a deviation (or no deviation) of an actualplacement of at least one product from the desired placement of productsassociated with the second planogram is present, as described elsewherein this disclosure. If computing device 1801 determines that a deviationof an actual placement of at least one product from the desiredplacement of products associated with the second planogram is present,computing device 1801 may issue a user-notification indicating thedeviation (e.g., at least one product is located out of place), asdescribed elsewhere in this disclosure.

In some embodiments, a third set of images may be captured at a thirdtime after the determination of whether the arrangement associated withthe second plurality of products conforms to the second planogram oranother planogram (e.g., a third planogram). The images may be capturedby a capturing device as described elsewhere in this disclosure. In someembodiments, the third set of images may depict a third plurality ofproducts displayed on the shelves of the retail store. Computing device1801 may be configured to receive the third set of images and analyzethe third set of images to determine the actual place of products on theshelves of the retail store at the third time, as described elsewhere inthis disclosure. Computing device 1801 may also be configured todetermine a deviation (or no deviation) of an actual placement of atleast one product from the desired placement of products associated withthe second planogram or another planogram (e.g., a third planogram) ispresent, as described elsewhere in this disclosure. Computing device1801 may further be configured to compare the actual placement of thethird plurality of products as identified in the third set of images toa desired placement of products associated with the first planogramComputing device 1801 may also be configured to determine whether anarrangement associated with the third plurality of products conforms tothe first planogram based on the comparison. If computing device 1801determines that the arrangement associated with the third plurality ofproducts conforms to the first planogram, computing device 1801 may beconfigured to withholding issuance of a user-notification associatedwith the deviation of the actual placement of the at least one productfrom the desired placement of products associated with the secondplanogram.

An apparatus may be used to secure a camera, e.g., capturing device 125,to a retail shelving unit. The apparatus may enable traditional shelvingunits to be modified to support one or more capturing devices toimplement the systems and methods described herein. For example,embodiments of the disclosed apparatus may be used to mount capturingdevices to new or existing shelving units. Disclosed embodiments of theapparatus may be configured to be mounted on a support element of ahorizontal shelf and may enable capturing devices to be mounted tohorizontal shelves in a variety of configurations.

FIGS. 21A and 21B illustrate, in perspective view and side view,respectively, an embodiment of an exemplary apparatus 2100 for securinga camera to a retail shelving unit. The apparatus 2100 may include apair of opposing securing surfaces 2102 a and 2102 b, where the pair ofopposing securing surfaces form a gap 2104 between the pair of opposingsecuring surfaces. In one example, the width of the gap 2104 between thepair of opposing securing surfaces 2102 a and 2102 b may be between 1.5min to 8 mm. In one example, the width of the gap 2104 between the pairof opposing securing surfaces 2102 a and 2102 b may be less than 1.5 mm.In one example, the width of the gap 2104 between the pair of opposingsecuring surfaces 2102 a and 2102 b may be wider than 8 mm. The gap 2104may be configured to accept the support element of a retail shelvingunit, as described further below. In some embodiments, the width of gap2104 may be similar to the width of the support element that the gap isconfigured to receive such that the support element fits snugly into thegap. In some examples, the width of gap 2104 may be wider (for example,less than 1 mm wider, between 1 mm and 4 mm wider, at least 4 mm wider,etc.) than the width of the support element that the gap is configuredto receive.

In some embodiments, opposing securing surfaces 2102 a and 21021 mayfurther include one or more holes or openings 2114 configured to receivea fastener, e.g., a screw, nail, bolt, etc., for securing the apparatus2100 to a support element of a shelving unit. For example, securingsurface 2102 a may include no such hole, exactly one such hole, exactlytwo such holes, at least three such holes, and so forth. In anotherexample, securing surface 2102 b may include no such hole, exactly onesuch hole, exactly two such holes, at least three such holes, and soforth. In one example, opposing securing surfaces 2102 a and 2102 b mayinclude the same number of holes 2114. In another example, opposingsecuring surfaces 2102 a and 2102 b may include a different number ofholes 2114. In some examples, opposing securing surfaces 2102 a and 2102b may include opposing holes 2114. In some examples, opposing securingsurfaces 2102 a and 2102 b may include holes not opposing each other. Inother embodiments, securing surfaces 2102 a and 2102 b may be configuredto snap-fit onto a support element.

Apparatus 2100 may further include an arm 2106 extending from one of thepair of opposing securing surfaces (e.g., surface 2102 b). The arm 2106may have a distal end 2108 including a camera support area 2110including at least one opening 2112. Camera support area 2110 may beconfigured to receive a camera, e.g., capturing device 125. For example,camera support area 2110 may define a planar base against which a cameramay be mounted. Camera support area 2110 may be positioned such that itforms an orthogonal angle with respect to securing surfaces 2102 a and2102 b. For example, camera support area 2110 may be parallel with aside of a shelving unit facing an aisle.

In some embodiments, the camera support area 2110 may include a hole oropening 2112 for placement of a cable connecting a camera to a batterysource. For example, opening 2112 may enable a power source to beconnected to a camera with a power cord. Opening 2112 may be furtherconfigured to receive one or more cords or connections. The one or morecords may include, for example, a data conduit, e.g., data conduit 508shown in FIG. 6B, an Ethernet cable, a power cord, etc. In anotherembodiment, opening 2112 may be configured to receive a fastener, e.g.,screw, nail, bolt, etc. for securing the camera to the camera supportarea.

FIG. 21C illustrates another embodiment of an apparatus 2116. Apparatus2116 may include opposing securing surfaces 2118 a and 2118 b. An arm2120 may extend from a securing surface 2118 b and may include a camerasupport area 2122. Camera support area 2122 may be positioned such thatthe plane defined by the camera support area is orthogonal to a planedefined by a front edge of a horizontal shelf. The plane defined bycamera support area 2122 may also be orthogonal to a plane defined bysecuring surfaces 2118 a and 2118 b. In some embodiments, opposingsecuring surfaces 2118 a and 2118 b may further include one or moreholes or openings configured to receive a fastener, e.g., a screw, nail,bolt, etc., for securing the apparatus 2116 to a support element of ashelving unit. For example, securing surface 2118 a may include no suchhole, exactly one such hole, exactly two such holes, at least three suchholes, and so forth. In another example, securing surface 2118 b mayinclude no such hole, exactly one such hole, exactly two such holes, atleast three such holes, and so forth. In one example, opposing securingsurfaces 2118 a and 2118 b may include the same number of holes. Inanother example, opposing securing surfaces 2118 a and 2118 b mayinclude a different number of holes. In some examples, opposing securingsurfaces 2118 a and 2118 b may include opposing holes. In some examples,opposing securing surfaces 2118 a and 2118 b may include holes notopposing each other. In other embodiments, securing surfaces 2118 a and2118 b may be configured to snap-fit onto a support element.

FIG. 21D is another embodiment of an apparatus 2124. Apparatus 2124 mayinclude opposing securing surfaces 2126 a and 2126 b and an arm 2128extending from securing surface 2126 b. The arm 2128 may include acamera support area 2130, which is parallel to a direction of thesecuring surfaces 2126 a and 2126 b. Camera support area 2130 mayinclude one or more openings 2132 configured to receive one or morecables for connecting a camera to an electronic component or powersource. Openings 2132 may also be configured to receive a fastener tomount a camera to apparatus 2124,

The exemplary apparatuses, 2100, 2116, and 2124, shown in FIGS. 21A-21Dmay each be formed of a continuous, rigid material. For example,apparatus 2100 may be formed of a rigid plastic through an injectionmolding or 3D printing process. In another embodiment, the pair ofopposing securing surfaces and the arm may be constructed from a singlecontinuous galvanized metal sheet. In one example, the thickness of thecontinuous galvanized metal sheet may be between 1 mm and 3 mm. In oneexample, the thickness of the continuous galvanized metal sheet may beless than 1 mm. In one example, the thickness of the continuousgalvanized metal sheet may be more than 3 mm. In some embodiments, asshown in FIG. 21A, the continuous galvanized metal sheet may include atleast one bend 2134 greater than 90 degrees along at least oneorthogonal rotation axis (for example, greater than 150 degrees, greaterthan 170 degrees, less than 180 degrees, less than 200 degrees, and soforth). The angle may be based on, for example, the desired area to becaptured by the lens of capturing device 125. For example, as shown inFIGS. 21A, 21C, and 21D, different configurations may enable camerasmounted to each of apparatus 2100, 21146, and 2124 to capture differentviews of an aisle or horizontal shelf.

In some embodiments, as shown in FIGS. 21A and 21C, the at least onebend of the continuous galvanized metal sheet may form two u-shapedstructures. These u-shaped structures may include a first u-shapedstructure that is associated with the pair of opposing securing surfacesand a second u-shaped structure that is associated with a surfaceassociated with the camera support area.

In some embodiments, a first type of apparatus (e.g., apparatus 2100)may have an arm 2106 associated with a first length, l₁, and a secondtype of apparatus (e.g., apparatus 2116) may have an arm 2120 associatedwith a second length, L₂. In some embodiments, the second length, l₂,may be greater than the first length, l₁. In other embodiments, l₁ andl₂ may be equal. In yet another embodiment, l₁ may be less than l₂. Thelength of arm 2106, extending from one of the pair of opposing securingsurfaces, may be between 30 mm and 70 mm, may be less than 30 mm, may bemore than 70 mm, and so forth. In some embodiments, the length, l_(ss),of each securing surface may be between 30 mm and 70 mm, may be lessthan 30 mm, may be more than 70 mm, and so forth.

FIG. 22A illustrates a method for attaching an apparatus 2100 to ahorizontal shelf 2202 of a retail shelving unit. Shelf 2202 may besupported by a support element 2204. In some embodiments, supportelement 2204 may be a rigid piece of metal or plastic configured tosupport horizontal shelf 2202. In one example, support element 2204 andshelf 2202 may form a continuous, rigid material, while in anotherexample, support element 2204 and shelf 2202 may be separate parts,which in some examples may be connected together (for example, usingglue, using fastener, using screws, using nails, using bolts, and soforth).

Apparatus 2100 may be joined to the retail shelving unit by insertingsupport element 2204 into the gap formed by the opposing securingsurfaces. The pair of opposing securing surfaces may be configured to befixedly mounted to the support element, wherein the support elementincludes a rigid structural element supporting a horizontal shelf of theretail shelving unit. In some embodiments, shelf support 2204 may fitsnugly between opposing securing surfaces, e.g., surfaces 2102 a and2102 b. In other embodiments, apparatus 2100 may be further joined tothe shelving unit using a number of fastening methods including screws,nails, glue, etc., disposed in holes 2114.

FIGS. 22B illustrates a camera, e.g., capturing device 125, mounted onshelf 2202 using apparatus 2100. The camera may be mounted on camerasupport area 2110 and may be positioned such that a lens of the camerafaces away from shelf 2202. In some embodiments, as shown in FIG. 22B,the apparatus is configured such that when the apparatus is fixedlymounted to the shelf, at least a portion of the apparatus is locatedbetween two horizontal shelves. In some embodiments, the apparatus maybe configured such that when the apparatus is fixedly mounted to theshelf, at least a portion of the apparatus, e.g., one or the securingsurfaces and/or arm, is located between two horizontal shelves. Thewidth of the securing surface may be less than a space between the twohorizontal shelves such that one of the securing surfaces fits betweenthe shelves. In some embodiments, the width of the planar base, e.g.,camera support area 2110 may be greater than a space between the twohorizontal shelves.

Apparatuses 2100, 2116, and 2124 may be used to mount cameras as part ofa system for monitoring one or more shelves and/or aisles, e.g., in astore such as a grocery store. In some embodiments, such a system mayinclude a plurality of cameras, e.g., capturing devices 125, configuredto capture images depicting a plurality of products displayed in theretail store. Each camera may be mountable to a retail shelving unitusing an apparatus, e.g., apparatus 2100. As previously described, theapparatus may include a pair of opposing securing surfaces for securingthe apparatus to a support element of the retail shelving unit and anarm extending from one of the pair of opposing securing surfaces andhaving a distal end. The distal end may include a camera support areaincluding at least one opening. The system may further include a datainterface configured to receive image data from the plurality of camerasand at least one processor for monitoring planogram compliance using thereceived image data. For example, as shown in FIGS. 1 and 2, imageprocessing unit 130 may include a server 135 configured to communicatewith one or more capturing devices 125 via network interface 206.Network 150 may facilitate data exchange between system components,e.g., image processing unit 130 and capturing device 125, therebyenabling capturing devices 125 to output image data to image processingunit 130. Image processing unit 130 may include a processor, e.g.,processor 202, configured to execute instructions for monitoringplanogram compliance using the received image data.

In some embodiments, the system may include a plurality of apparatusesthat includes at least two types of apparatuses for mounting theplurality of cameras to a plurality of retail shelving units. Anapparatus from a first type may be configured such that when it isfixedly mounted to a support element of a horizontal shelf, its camerasupport area is on a surface positioned substantially parallel to a sideof the horizontal shelf facing an aisle. An apparatus from a second typemay be configured such that when it is fixedly mounted to a supportelement of a horizontal shelf, its camera support area is on a surfacepositioned substantially orthogonal to the pair of opposing securingsurfaces.

In some embodiments, the system may include a plurality of apparatusesthat includes at least two types of apparatuses for mounting theplurality of cameras to a plurality of retail shelving units. The atleast two types of apparatus may include a first type having an armassociated with a first length and a second type having an armassociated with a second length greater than the first length.

FIG. 23 illustrates an exemplary system in which a number of imagecapture devices are mounted to retail shelving unit 2302. An apparatusof a first type (e.g., apparatus 2100) may be configured such that whenit is fixedly mounted to a support element 2304 of a horizontal shelf2306, its camera support area 2110 defines a surface positionedsubstantially parallel to a front side 2308 of the horizontal shelffacing an aisle 2310. For example, when a camera 2314 is mounted tosupport area 2110, it faces toward an opposite shelf 2312 across aisle2310. Thus, the camera 2314 may capture images of shelf 2312 and/orproducts on shelf 2312. In some embodiments, camera support, area 2110may define a surface positioned parallel to a front side 2308 of thehorizontal shelf. For example, the apparatus may be configured such thatwhen the apparatus is fixedly mounted to a support element of ahorizontal shelf, the planar base is positioned parallel orsubstantially parallel to a side of the horizontal shelf facing anaisle.

An apparatus of a second type (e.g., apparatus 2116) may be configuredsuch that when it is fixedly mounted to a support element of ahorizontal shelf 2316, its camera support area 2122 is on a surfacepositioned substantially orthogonal to the pair of opposing securingsurfaces 2118 a and 2118 b. For example, camera 2318 may be mounted toshelf 2316 using apparatus 2116 such that camera 2318 is oriented toface downward. Thus, camera 2318 may be configured to capture images ofthe front of shelving unit 2302 or of items on lower shelf 2306. In someembodiments, camera support area 2122 may define a surface positionedorthogonal to the pair of opposing securing surfaces 2118 a and 2118 b.

An apparatus of a third type (e.g., apparatus 2124) may be configuredsuch that when it is fixedly mounted to a support element of ahorizontal shelf 2316, its camera support area 2130 is on a surfacepositioned substantially parallel to the pair of opposing securingsurfaces 2126 a and 2126 b. In some embodiments, camera support area2130 may be on a surface positioned parallel to the pair of opposingsecuring surfaces 2126 a and 2126 b. The apparatus 2100 may beconfigured such that when the apparatus is fixedly mounted to thesupport element of a horizontal shelf, the planar base, e.g., camerasupport area 2130, is positioned substantially orthogonal to a side ofthe horizontal shelf facing an aisle, e.g., aisle 2310. In someembodiments, camera support area 2130 may be positioned orthogonal to aside of the horizontal shelf facing an aisle.

For example, camera 2320 may be mounted to shelf 2316 using apparatus2124 such that camera 2320 is oriented to face along the front ofshelving unit 2302. Thus, camera 2320 may be configured to captureimages of the aisle 2310 between shelving units 2302 and 2312.

In some embodiments, multiple apparatuses may be used to secure camerasto different shelves of different types. For example, the firstapparatus may be attachable to a first horizontal shelf (e.g., shelf2306) and the second apparatus may be attachable a second horizontalshelf (e.g., shelf 2316). The first horizontal shelf and the secondhorizontal shelf may be of different heights or the first horizontalshelf and the second horizontal shelf may of the same height. In anotherexample, a first horizontal shelf may be attachable to a firsthorizontal shelf and a second apparatus may be attachable to a secondhorizontal shelf, and the first horizontal shelf and the secondhorizontal shelf may be of different sizes.

In some embodiments, e.g., as shown in FIGS. 6A and 6B, at least two orthe plurality of cameras may be mountable on a single horizontal shelfand configured to be connected by a cable configured to covey power andimage data. For example, a cable may extend between multiple capturingdevices via opening 2112 in apparatus 2100. In some embodiments, thecable may be located behind a plastic piece configured to cover the edgeof the shelf opposing the aisle. For example, the plastic piece may bemounted on front-faring surface 2308 of a shelf 2306.

The above described system 2300 may be configured to execute one or moremethods or processes as described herein. For example, image capturedevices mounted using apparatuses 2100, 2116, and 2124 may be used tocapture data regarding customers and/or products in an environment ofsystem 2300.

In some embodiments, a system may process images captured in a retailstore and automatically determine relationships between products andshelf labels. The system may be a system, e.g., system 100, including atleast one processor, e.g., image processing unit 130. The system mayfurther include one or more image capture devices, e.g., capturingdevices 125. In some examples, processing images from a single point intime may be insufficient to determine the relationships between productsand shelf labels. In one example, the visual information on the shelflabels and/or product may be insufficient to determine the relationships(for example, the shelf label may include no visual identifier of theproduct). In one example, the product may be unfamiliar to the system(for example, new product, product with a new visual appearance, and soforth). In one example, the visual information in the images may beinsufficient to determine the relationships between products and shelflabels, for example due to low pixel resolution, image blurring,occlusions, and so forth. In one example, the system product recognitionengine and/or label recognition engine may lack the ability to recognizethe product and/or the shelf label for various reasons, and as a resultthe system may be unable to determine the relationships between productsand shelf labels from an analysis of the images from the single point intime. In some embodiments, analysis of movements of the locations of theproduct and/or the shelf label may be used to determine the relationshipor the lack of relationship between the product and the shelf label.

In some examples, a shelf label may be positioned adjacent to two ormore products (in this case, a product may refer to a plurality ofproduct instances that have a common product type, the product type maybe known or unknown) at a first point in time. Further, in someexamples, the position of at least one of the two or more products maychange between the first point in time and a second point in time. Inone example, in response to a movement of the shelf label that issimilar to the movement of a particular product (i.e., in the samedirection, at the same time, to the same retail shelf, etc.) between thefirst point in time and the second point in time, it may be determinedthat the shelf label corresponds to the particular product. In oneexample, in response to no movement of the shelf label (or nosignificant movement, such as movement of a distance that is below aselected threshold, no movement between retail shelves, etc.) betweenthe first point in time and the second point in time, it may bedetermined that the shelf label corresponds to a product of the two ormore products that did not move (or did not have a significant change inposition, such as movement of a distance that is below a selectedthreshold, no movement between retail shelves, etc.) between the firstpoint in time and the second point in time. In one example, in responseto a movement of the shelf label that is not similar to the movement ofa particular product (i.e., in a different direction, at a differenttime, to a different retail shelf, etc.) between the first point in timeand the second point in time, it may be determined that the shelf labeldoes not correspond to the particular product.

In some examples, two or more shelf labels may be positioned adjacent toa product (in this case, a product may refer to a plurality of productinstances that have a common product type, the product type may be knownor unknown) at a first point in time. Further, in some examples, theposition of at least one of the two or more shelf labels may changebetween the first point in time and a second point in time. In oneexample, in response to a movement of the product that is similar to themovement of a particular shelf label of the two or more shelf labels(i.e., in the same direction, at the same time, to the same retailshelf, etc.) between the first point in time and the second point intime, it may be determined that the product corresponds to theparticular shelf label. In one example, in response to no movement ofthe product (or no significant movement, such as a movement of adistance that is below a selected threshold, no movement between retailshelves, etc.) between the first point in time and the second point intime, it may be determined that the product corresponds to a shelf labelof the two or more products that did not move (or did not have asignificant change in position, such as a movement of a distance that isbelow a selected threshold, no movement between retail shelves, etc.)between the first point in time and the second point in time. In oneexample, in response to a movement of the product that is not similar tothe movement of a particular shelf label of the two or more shelf labels(i.e., in a different direction, at a different time, to a differentretail shelf, etc.) between the first point in time and the second pointin time, it may be determined that the product does not correspond tothe particular shelf label.

In some examples, a shelf label may be positioned adjacent to a product(in this case, a product may refer to a plurality of product instancesthat have a common product type, the product type may be known orunknown) at a first point in time. In one example, in response to amovement of the product and the shelf label in a similar way (i.e., inthe same direction, at the same time, to the same retail shelf, etc.)between the first point in time and a second point in time, it may bedetermined that the product corresponds to the particular shelf label.In on example, in response to a movement of the shelf label and nomovement of the product (or in response to a movement of the shelf labeland no significant movement of the product, such as a movement of adistance that is below a selected threshold, no movement between retailshelves, etc.) between the first point in time and the second point intime, it may be determined that the product does not correspond to theshelf label. In on example, in response to a movement of the product andno movement of the shelf label (or in response to a movement of theproduct and no significant movement of the shelf label, such as amovement of a distance that is below a selected threshold, no movementbetween retail shelves, etc.) between the first point in time and thesecond point in time, it may be determined that the product does notcorrespond to the shelf label. In one example, in response to a movementof the product that is not similar to the movement of the shelf label(for example in a different direction, at a different time, to adifferent retail shelf, etc.) between the first point in time and thesecond point in time, it may be determined that the product does notcorrespond to the shelf label.

In some example, analysis of movements of the locations of the product(in this case, a product may refer to a plurality of product instancesthat have a common product type, the product type may be known orunknown) and/or the shelf label may be combined together with other cuesto determine the relationship or the lack of relationship between theproduct and the shelf label. Some non-limiting examples of such othercues may include information based on an analysis of the visualappearance of the shelf label and/or the product, information based onan analysis of textual information appearing on the shelf label (such asproduct name, product size, price, and so forth) and/or the product(such product name, product size, and so forth), information based onthe locations of the product and the shelf label, and so forth. Forexample, analysis of the movement of the locations of the product and/orthe shelf label may suggest a first confidence level associated with arelation of the product and the shelf label, other cues (for example, asdescribed above) may suggest a second confidence level associated withthe relation of the product and the shelf label, and the first and thesecond confidence levels may be combined to determine whether theproduct corresponds to the shelf label or not. For example, each one ofthe first confidence level and the second confidence level may include anumerical value, the two values may be combined using a mathematicalfunction (such as sum, mean, median, non-linear function, and so forth),and the combination of the first and second values may be compared witha selected threshold to determine whether the product corresponds to theshelf label or not.

In some examples, a machine learning model may be trained using trainingexamples to determine locations of shelf labels from images and/orvideos, and the trained machine learning model may be used to analyze animage of a store shelf to determine position of shelf labels on thestore shelf. An example of such training example may include an imageand/or a video of the store shelf, together with a label indicating thelocation of shelf labels on the store shelf. In some examples, a machinelearning model may be trained using training examples to determinelocations of products from images and/or videos, and the trained machineteaming model may be used to analyze an image of a store shelf todetermine position of products on the store shelf. An example of suchtraining example may include an image and/or a video of the store shelf,together with a label indicating the location of products on the storeshelf. In some examples, images of at least one store shelf capturedover a period of time by one or more image sensors may be analyzed, forexample as described above, to identify positions of shelf labels and/orproducts on the store shelf at different point in time. In someexamples, shelf labels and/or products identified on the store shelf atdifferent points in time may be matched (for example, based on visualappearance, based on textual information appearing on the shelf labelsand/or products, and so forth) to determine movement of shelf labelsand/or products between different point in time (for example, between afirst point in time and a second point in time). For example, shelflabels (or products) with the same particular text (such as a particularprice, a particular product name, etc,) and/or the same particular logomay be identified in two different locations at the two points in time,and may be matched based on the particular text and/or the particularlogo. In another example, a machine learning model may be trained usingtraining examples to match shelf labels in pairs of images, and thetrained machine learning model may be used to analyze an image of thestore shelf at a first point in time and an image of the shelf at asecond point in time to match shelf labels in the two images. An exampleof such training example may include a pair of images, together with alabel indicating matches between shelf labels in the two images, in yetanother example, a machine learning model may be trained using trainingexamples to match products in pairs of images, and the trained machinelearning model may be used to analyze an image of the store shelf at afirst point in time and an image of the shelf at a second point in timeto match products in the two images. An example of such training examplemay include a pair of images, together with a label indicating matchesbetween products in the two images.

In some examples, the movements of the locations of the product (in thiscase, a product may refer to a plurality of product instances that havea common product type, the product type may be known or unknown) and/orthe shelf label may be analyzed to determine the relationship or thelack of relationship between the product and the shelf label, and/or todetermine a confidence level associated with the relationship. In oneexample, a movement may be represented as a mathematical vector (forexample, by subtracting the location at the first point in time from thelocation at the second point in time, by embedding the movement in amathematical space, etc.), a correlation between the vector representingthe movement of the product and the vector representing the movement ofthe shelf label may be calculated, and the calculated correlation may beused to determine the relationship or the lack of relationship betweenthe product and the shelf label, and/or to determine a confidence levelassociated with the relationship (for example, based on a comparison ofthe correlation with a selected threshold, by calculating a function ofthe correlation, and so forth). In another example, a machine learningmodel may be trained using training examples to determine therelationship or the lack of relationship between products and the shelflabels (and/or to determine confidence level associated with therelationships) from the movements of the locations of the productsand/or the shell labels, and the trained machine learning model may beused to analyze a determined movement of a particular shelf label and adetermined movement of a particular label to determine a relationship orthe lack of relationship between the particular product and theparticular shelf label (and/or to determine confidence level associatedwith the relationship). An example of such training example may includean indication of a movement of a shelf label and an indication of amovement of a product, together with a label indicating whether theproduct correlates to the shelf label or not.

The system may be configured to receive images of at least one storeshelf captured over a period of time by one or more image sensors. Forexample, image processing unit 130 may receive images and/or image datafrom capturing devices 125 that may be placed in any location in aretail location. As shown in FIG. 4A, a number of capturing devices 125may be mounted on a retail shelving unit and be configured to captureimages of an opposing shelving unit. For example, a capturing device 125may be configured to collect images associated with products or labelson an opposing shelf. Thus, the collected images may depict a pluralityof shelf labels and groups of products associated with different producttypes placed on the at least one store shelf.

The system may analyze the images to identify temporal changes in anarrangement of the groups of products over the period of time. Forexample, processing unit 130 may receive a time series of imagescaptured during a period or time, e.g., over one minute, one hour, oneday, etc. The images may depict, at a first point in time, the groups ofproducts are arranged on the least one store shelf in a first manner.For example, the first time may be the time the retail location opens tocustomers. The group of products may be arranged on a shelf in anordered manner. The images may also depict, at a second point in time,the groups of products are arranged on the least one store shelf in asecond manner. For example, at a later time, the group of products maybe scattered, e.g., after one or more customers have selected productsfrom the shelf, in another example, one of the products may be placed inan incorrect location, e.g., near a label associated with a differentproduct.

Processing unit 130 may also analyze the images to identify temporalchanges in display of the plurality of shelf labels over the period oftime. The images may depict, at the first point in time, the pluralityof shelf labels are displayed on the least one store shelf in a firstmanner. The images may also depict, at the second point in time, theplurality of shelf labels are displayed on the least one store shelf ina second manner. For example, between the two points in time, the shelfmay have been restocked or reorganized by an employee. Alternatively oradditionally, shelf labels may have been moved inadvertently, or labelsmay have been added or removed according to stocking instructions.

Processing unit 130 may then determine a relationship between a specificgroup of products and a specific shell label based on the identifiedtemporal changes in the arrangement of the groups of products and theidentified temporal changes in the display of the plurality of shelflabels. Based on this relationship, processing unit 130 may initiate anaction based on the determined relationship.

In some embodiments, processing unit 130 may transmit informationrelated to the action to a device associated with a retail store,supplier, or market research entity. For example, processing unit 130may transmit instructions to a user device causing the user device todisplay the determined action. In another embodiment, processing unit130 may transmit instructions to an autonomous robot configured to causethe robot to complete the action.

FIG. 24 is a block diagram of a processing device 202 of system 100.Processing device 202 may include one or more components configured tocause the processing device 202 to process images captured in a retailstore and automatically determine relationships between products andshelf labels. Processing device 202 may include an image analysiscomponent 2402, a relationship determination component 2404, and anaction component 2406.

Image analysis component 2402 may include a memory storing softwareinstructions and/or modules configured to receive one or more imagesfrom capturing device 125 and analyze the images to determinearrangements of products and shelf labels on a retail shelving unit.Image analysis component 2402 may receive raw or processed data fromcapturing device 125 via respective communication links, and provideinformation to different system components using a network 150.Specifically, image analysis component 2402 may use any suitable imageanalysis technique including, for example, object recognition, objectdetection, image segmentation, feature extraction, optical characterrecognition (OCR), object-based image analysis, shape region techniques,edge detection techniques, pixel-based detection, artificial neuralnetworks, convolutional neural networks, etc. In addition, imageanalysis component 2402 may use classification algorithms to distinguishbetween the different products or labels in the retail store. In someembodiments, image analysis component 2402 may utilize suitably trainedmachine learning algorithms and models to perform the product and/orlabel identification. In some embodiments, the products and/or labelsmay include scannable codes (e.g., barcodes or QR codes).

With reference to FIG. 25A, a camera, e.g., capturing device 125, maycapture an image 2500 of a shelving unit 2502 at a first time, t₁.Capturing device 125 may be fixedly mounted to a retail shelving unitopposing the at least one store shelf, e.g., shelf 2502, may be mountedelsewhere in the retail store, may be part of a mobile device, may bepart of a robot configured to capture images, and so forth. Imageanalysis component 2402 may receive the image data from capturing device125, e.g,, via network 150, and may perform one or more image processingtechniques to identify products and/or labels within the received image.For example, image analysis component 2402 may identify a first product2504 arranged on a shelf 2506 above a first label 2508. Image analysiscomponent 2402 may also identify a second product 2510 arranged on ashelf 2512 above a second label 2514. Image analysis component 2402 mayfurther identify products 2516 and 2518, associated with labels 2520 and2522, respectively.

FIG. 25B is an illustration of a second image 2501 of shelf 2502 at asecond time, t₂. The second time may be at a point in time later thanfirst time t₁. In some embodiments, for example, the images of the atleast one store shelf are captured over a period of time greater than aday, e.g., a week or month. Image analysis component 2402 may analyzeimage 2501 to identify an arrangement of the products at t₂ and aplacement of labels at t₂. For example, image analysis component 2402may analyze the images from multiple points in time, e.g., t₁ and t₂,and identify one or more temporal changes between the two images. Thetemporal changes may include rearrangement of products, depletion ofproducts, addition of labels to the shelves, shift in product positionon the shelf, etc.

In some embodiments, the temporal changes in the arrangement of thegroups of products may include a depletion of products from the specificgroup of products from the at least one store shelf, and the temporalchanges in the display of the plurality of shelf labels may include aremoval of the specific shelf label from the at least one store shelf.For example, the temporal change between two time points may include adepletion of product 2518 and/or a removal of label 2522. In anotherexample, the temporal changes in the arrangement of the groups ofproducts may include an exhaustion of the specific group of products tothe at least one store shelf. For example, a depletion or an exhaustionmay occur if customers have purchased the product between the two timepoints and there has not yet been a restocking event. In yet anotherexample, the temporal changes in the arrangement of the groups ofproducts may include an addition of the specific group of products tothe at least one store shelf, and/or the temporal changes in the displayof the plurality of shelf labels may include an addition of the specificshelf label to the at least one store shelf.

In another example, the temporal changes in the arrangement of thegroups of products may include a removal of the specific group ofproducts from the at least one store shelf, and/or the temporal changesin the display of the plurality of shelf labels may include a removal ofthe specific shelf label from the at least one store shelf. For example,a store may sell out of product 2516, or the product may no longer bestocked by the store.

In another example, the temporal changes in the arrangement of thegroups of products may include a reposition of the specific group ofproducts to a different location on the at least one store shelf, and/orthe temporal changes in the display of the plurality of shelf labels mayinclude a reposition of the specific shelf label. For example, as shownin FIG. 25B, product 2504 and label 2508 have been relocated to a highershelf.

In another example, the temporal change may include the addition of agroup of products 2524 and/or an addition of label 2526. For example,between the two time points, a group of products may have been restockedon the shelf or may have been moved to the shelf from another locationin the store.

Returning to FIG. 24, relationship determination component 2404 mayinclude a memory storing software instructions and/or modules and mayreceive the analyzed image data and identified temporal changes fromimage analysis component 2402. Relationship determination component 2404may determine a relationship between a group of products on the shelfand a label. For example, relationship determination component 2404 maydetermine, based on the product and label information from imageanalysis component 2402, that products 2504 are associated with label2508. This determination may also be based on, for example, the distanceof the product group relative to the label in one or both images. Insome embodiments, the at least one processor is further configured torecord the determined relationship between the specific group ofproducts placed on the at least one store shelf and the specific shelflabel for future use. For example, the system may store relationshipinformation in database 140.

Relationship determination component 2404 may also, for example,identify a specific product type for the specific group of productsplaced on the at least one store shelf based on information depicted onthe specific shelf label. For example, a product type may be determinedbased on text of the label, indicating a product name, on a scannablecode (e.g., barcode) displayed on the label, on a price of the product,etc. Relationship determination component 2404 may determine therelationship between the specific group of products and the specificshelf label by recognizing products from the specific group of productsas products of a specific product type identified in the specific shelflabel.

In some embodiments, image analysis component 2402 may receive andanalyze additional images captured at a time later than the time period,e.g., after t₂. The additional images may depict additional productsplaced on the at least one store shelf. Image analysis component 2402may analyze the additional images to identify an existence of thespecific shelf label attached to the at least one store shelf. Based onthe analysis, relationship determination component 2404 may recognize agroup of the additional products as products from the specific producttype based on the existence of the specific shelf label.

Action component 2406 may include a memory storing software instructionsand/or modules configured to receive information from image analysiscomponent 2402 and relationship determination component 2404 anddetermine whether or not to initiate an action. For example, if aproduct associated with a label is depleted, action component 2406 mayinitiate a restock of the product. In another example, the action mayinclude issuing a user-notification about a misplaced shelf label to astore employee.

In some embodiments, the action may include updating a machine learningmodel for product recognition based on the determined relationshipbetween the specific group of products placed on the at least one storeshelf and the specific shelf label. For example, action component 2406may initiate the addition of relationship information to a dataset. Theinformation may be one or more images of products of the specific groupof products with labels determined according to visual information onthe specific label. The action may further include updating the machinelearning model using the updated dataset. In another example, machinelearning algorithms may use one or more images of products of thespecific group of products with labels determined according to visualinformation on the specific label to update the machine learning model(e.g., without updating a dataset).

In some embodiments, action component 2406 may determine to forgo anaction based on information received from image analysis component 2402and relationship determination component 2404. For example, an actionmay be forgone if a label is placed in proximity to the specific groupof products, or the addition of the specific group of products and thespecific label occur at the same time or at close by times.

For example, the system may forgo initiating the action when thetemporal changes in the display of the plurality of shelf labelsincludes repositioning of the specific shelf label, and the temporalchanges in the arrangement of the groups of products do not include arepositioning of the specific group of products to a different locationon the at least one store shelf.

In another example, when the temporal changes in the arrangement of thegroups of products do not include running out of products from thespecific group of products from the at least one store shelf, and thetemporal changes in the display of the plurality of labels includes aremoval of the specific label, action component 2406 may forgoinitiating the action. In other cases, initiating an action may beforgone when the temporal changes in the arrangement of the groups ofproducts includes running out of products from the specific group ofproducts from the at least one store shelf, and the temporal changes inthe display of the plurality of labels does not include a removal of thespecific label.

In another example, the temporal changes in the arrangement of thegroups of products does not include an addition of the specific group ofproducts to the at least one store shelf, and the temporal changes inthe display of the plurality of labels includes an addition of thespecific label. Based on this temporal change, action component 2406 maydetermine to forgo initiating an action, e.g., if the added label is anindicator of a sale on a displayed product. In another example, noaction may be initiated when the temporal changes in the arrangement ofthe groups of products include an addition of the specific group ofproducts to the at least one store shelf, and the temporal changes inthe display of the plurality of labels does not include an addition ofthe specific label. In this example, the shelf may have been restockedand no further action is required.

In another example, an action may not be initiated by action component2406 when the temporal changes in the arrangement of the groups ofproducts does not include a removal of the specific group of productsfrom the at least one store shelf, and the temporal changes in thedisplay of the plurality of labels includes a removal of the specificlabel. When the temporal changes in the arrangement of the groups ofproducts includes a removal of the specific group of products from theat least one store shelf, and the temporal changes in the display of theplurality of labels does not include a removal of the specific label,action component 2406 may determine not to initiate an action.

In some embodiments, action component 2406 may determine at least onevalue associated with the determined relationship between the specificshelf label and the specific group of products placed on the at leastone store shelf; and initiating the action when the at least one valueis greater than a threshold. As previously described, relationshipdetermination component 2404 may determine a spatial relationship of aproduct and its associated shelf label. Image analysis component 2402may determine, for example, an increase in the space between the productand its shelf label. If this space increases beyond a threshold, e.g.,six inches, set for the particular product, action component 2406 mayinitiate an action, e.g., reorganization of the shelf.

Thus, when the movement specific group of products has spatialcorrelation (e.g., same direction of movement, similar distance ofmovement) to the movement of the specific label beyond an electedthreshold, in the system may initiate an action. In another example, ifthe spatial correlation is below a threshold, e.g., the label andproduct are less than one inch apart, action component 2406 maydetermine not to initiate an action.

In another example, no action may be initiated when the temporal changesin the arrangement of the groups of products includes a reposition ofthe specific group of products to a different location on the at leastone store shelf and the temporal changes in the display of the pluralityof labels does not include a reposition of the specific label. When thetemporal changes in the arrangement of the groups of products does notinclude a reposition of the specific group of products to a differentlocation on the at least one store shelf and the temporal changes in thedisplay of the plurality of labels does not include a reposition of thespecific label, action component 2406 may not initiate an action. Inother cases, when the temporal changes in the arrangement of the groupsof products does not include a reposition of the specific group ofproducts to a different location on the at least one store shelf and thetemporal changes in the display of the plurality of labels includes areposition of the specific label, action component 2406 may not initiatean action.

In some embodiments, the system may analyze the images to identifytemporal changes in placements of a plurality of promotions associatedwith the at least one store shelf over the period of time. For example,image analysis component 2402 may analyze one or more labels on a shelfto determine a particular label is associated with a promotion, e,g.,based on the text of the label. Relationship determination component2404 may determine a second relationship between the specific producttype and a specific promotion based on the identified temporal changesin the placements of the plurality of promotions. Based on this secondrelationship, action component 2406 may initiate a second action. Thesecond action may include, for example, recording the specific promotionin association with the specific product in a database, issuing auser-notification regarding another shelf associated with the specificproduct that do not have a promotion similar or identical to thespecific promotion, issuing a user-notification that the specificpromotion have to be removed as no promotion is planned for the specificproduct, and so forth.

FIG. 26 is a flowchart of a method 2600 for processing images capturedin a retail store and automatically determining relationships betweenproducts and shelf labels. Method 2600 may be implemented by ageneral-purpose computer or a special-purpose computer built accordingto embodiments of the present disclosure. For example, method 2600 maybe executed by at least one processor, e.g., processing unit 130.

At step 2602, the system, e.g., image analysis component 2402, mayreceive images of at least one store shelf captured over a period oftime by one or more image sensors. For example, image sensors mayinclude capturing devices 125, which are mounted to a retail shelvingunit and configured to capture images of opposing shelving units. Theperiod of time may be one hour, one day, one week, etc.

At step 2604, image analysis component 2402 may analyze the images toidentify temporal changes in arrangement of the groups of products overthe period of time, wherein at a first point in time the groups ofproducts are arranged on the least one store shelf in a first manner andat a second point in time the groups of products are arranged on theleast one store shelf in a second manner.

At step 2606, image analysis component 2402 may analyze the images toidentify temporal changes in display of the plurality of shelf labelsover the period of time, wherein at the first point in time theplurality of shelf labels are displayed on the least one store shelf ina first manner and at the second point in time the plurality of shelflabels are displayed on the least one store shelf in a second manner.

For example, in steps 2604 and 2606, image analysis component 2402 maydetermine one or more temporal changes associated with the placement ofproducts and/or labels. As shown in FIGS. 25A and 25B, temporal changesmay include the removal of products, movement of products and/or labels,depletion of products, etc.

At step 2608, relationship determination component 2404 may determine arelationship between a specific group of products placed on the at leastone store shelf and a specific shelf label based on the identifiedtemporal changes in the arrangement of the groups of products and theidentified temporal changes in the display of the plurality of shelflabels. For example, relationship determination component 2404 maydetermine if a product is associated with a particular label. In someembodiments, relationship determination component 2404 may determinewhether a product is placed within a predetermined threshold distance ofa label.

At step 2610, action component 2406 may initiate an action based on thedetermined relationship. As previously discussed, the action may be arestock or reorganization action. Instructions for the action may betransmitted to a device associated with a store employee or manager. Inother embodiments, the action may be to alert a customer, via a userdevice of the customer, of a promotion associated with a product.

Method 2600 may further include determining a confidence levelassociated with the determined relationship between the specific shelflabel and the specific group of products placed on the at least onestore shelf. In some embodiments, when the confidence level is lowerthan or equal to a first threshold, action component 2406 may notinitiate an action.

At a subsequent time, after forgoing initiating the action, imageanalysis component 2402 may receive additional images captured at a timelater than the time period and may analyze the additional images todetermine an updated confidence level associated with the determinedrelationship between the specific group of products and the specificshelf label. Based on the updated confidence level, action component2406 may initiate the action when the updated confidence level isgreater than a second threshold. In some embodiments, the action may beforgone if the updated confidence level is equal to the first confidencelevel.

In some embodiments, a system may process images captured in a retailstore and automatically identify situations for which to withholdplanogram incompliance notifications. The system may be a system, e.g.,system 100, including at least one processor, e.g., image processingunit 130. The system may further include one or more image capturedevices, e.g., capturing devices 125.

The system may be configured to receive image data from a plurality ofimage sensors mounted in the retail store, wherein the image datadepicts a plurality of products displayed on at least one store shelf.For example, image processing unit 130 may receive images and/or imagedata from capturing devices 125 that may be placed throughout a retaillocation. As shown in FIG. 4A, a number of capturing devices 125 may bemounted on a retail shelving unit and be configured to capture images ofan opposing shelving unit. For example, a capturing device 125 may beconfigured to collect images associated with products displayed on anopposing shelf.

Image processing unit 130 may analyze the image data to identify atleast one product type associated with the plurality of productsdisplayed on at least one store shelf and to determine a placement ofproducts of the at least one product type on the at least one storeshelf. For example, processing unit 130 may analyze the images capturedby capturing device 125 to determine that a product belongs to aparticular type of grouping. Exemplary product types may includeperishables, non-perishables, produce, dry goods, etc.

The system may then access at least one planogram describing a desiredplacement of products of the at least one product type on shelves of aretail store. For example, the system may query a database, e.g.,database 140, storing one or more planograms associated with shelvingunits of the retail store. The planogram may be a graphical depiction ofthe placement of items and shelf labels on a shelving unit. Aspreviously described, a planogram may be associated with contractualobligations and/or other preferences related to the retailer methodologyfor placement of products on the store shelves. In some embodiments, theat least one planogram may be a realogram. The planogram may display orexplain where the products for the identified product type should belocated compared to other product types, on which store shelves theproducts should be places, in which area of the store the product shouldbe placed. etc.

The system may identify, based on the at least one planogram, adiscrepancy between the determined placement of products associated withthe identified at least one product type and the desired placement ofproducts associated with the identified at least one product type. Forexample, a discrepancy may be a product displayed on an incorrect shelf,a product displayed on a shelf having an incorrect label, etc. Thediscrepancy may further include a lack of product on the shelf, e.g., ifthe displayed product has sold out and requires restocking.

The system may determine whether a notice-override condition existsrelative to the identified at least one product type due tomisidentification of the product. For example, a depletion of productmay trigger a restocking notice. In another example, if a product ismisplaced, a notice may be triggered instructing a store employee ormanager that a shelf needs reorganization, if a notice-overridecondition is not determined to exist relative to the identified at leastone product type, the system may issue a user-notification associatedwith the identified discrepancy between the determined placement ofproducts and the desired placement of products. For example, thenotification may be transmitted, via network 150, to one or more userdevices associated with the retail store. The notification may includeinstructions for remedying the discrepancy between the shelf image andthe planogram.

In some embodiments, if a notice-override condition exists relative tothe identified at least one product type, the system may withholdissuance of a user-notification associated with the identifieddiscrepancy between the determined placement of products and the desiredplacement of products. For example, if a product of a certain type issold out, a notification instructing an employee to restock a shelf maybe overridden if a different product of the same product type isavailable within a threshold proximity of the sold out product.

FIG. 27 is a block diagram of a processing device 202 of system 100.Processing device 202 may execute one or more components (e.g., one ormore software modules) configured to cause the processing device 202 toprocess images captured in a retail store and automatically determinerelationships between products and shelf labels. Processing device 202may execute an image analysis component 2702, a discrepancydetermination component 2704, and a notice component 2706. Alternativelyor additionally, components 2702, 2704, and/ 2706 may be stored in amemory device (e.g., memory device 226).

Image analysis component 2702 may be configured to receive one or moreimages from capturing device 125 and analyze the images to determine aproduct type of a product arranged on a store shelf. Image analysiscomponent 2702 may be further configured to identify productarrangements and/or identify shelf labels associated with displayedproducts. Image analysis component 2702 may receive raw or processeddata from capturing device 125 via respective communication links, andprovide information to different system components using a network 150.Specifically, image analysis component 2702 may use any suitable imageanalysis technique including, for example, object recognition, objectdetection, image segmentation, feature extraction, optical characterrecognition (OCR), object-based image analysis, shape region techniques,edge detection techniques, pixel-based detection, artificial neuralnetworks, convolutional neural networks, etc. In addition, imageanalysis component 2702 may use classification algorithms to distinguishbetween the different products or labels in the retail store. In someembodiments, image analysis component 2702 may utilize suitably trainedmachine learning algorithms and models to perform the product and/orlabel identification and to determine product types. In someembodiments, the products and/or labels may include scannable codes(e.g., barcodes or QR codes) used to identify a product type or productassociated with a shelf label.

With reference to FIG. 28A, a camera, e.g., capturing device 125, maycapture an image 2800 of a shelving unit 2802. Capturing device 125 maybe fixedly mounted to a retail shelving unit opposing the at least onestore shelf, be mounted elsewhere in the retail store, be part of amobile device, be part of a robot configured to capture images, and soforth. Image analysis component 2702 may receive the image data fromcapturing device 125, e.g., via network 150, and may perform one or moreimage processing techniques to identify one or more products and theirassociated product types. For example, image analysis component 2702 mayidentify a first product 2804. Image analysis component 2702 maydetermine that product 2804 is of product type “pasta,” based on textrecognition of label 2806, which is within a threshold proximity ofproduct 2804. Image analysis component 2702 may use other techniques foridentifying a product type including text recognition of a productlabel, a determination of the type of products adjacent to the product,etc., for example as described above. In one example, a machine learningmodel may be trained using training examples to identify product typesfrom images and/or videos of products, and image analysis component 2702may use the trained machine learning model to analyze the image andidentify the type of the product. An example of such training examplemay include an image and/or a video depicting a product, together with alabel indicating the product type of the depicted product.

Returning to FIG. 27, image analysis component 2702 may analyze theimage, e.g., image 2800, and transmit the analyzed image data todiscrepancy detection component 2704. Discrepancy detection component2704 may receive the analyzed data and may also access one or moreplanograms associated with a product type determined to be present inimage 2800. For example, discrepancy detection component 2704 mayreceive product type information from image analysis component 2702.Based on this product type information, discrepancy detection component2704 may query a database, e.g., database 140, to retrieve one or moreplanograms associated with the identified product types. In someembodiments, planograms may be stored in a planogram database along withassociated product type tags. Thus, a user or system may query thedatabase using a product type and the database may return one or moreplanograms tagged with the particular product type.

FIG. 28B illustrates an exemplary planogram 2808 associated withshelving unit 2802. The planogram may be associated with shelving unit2802 based on the identification of “pasta” products on the shelf 2812in image 2800, based on a location of shelving unit 2802, based on avisual code identifying shelving unit 2802, based on nearby shelves, andso forth. In some embodiments, the planogram for shelf 2812 may be basedon the combination of product types on shelf 2812. As shown in FIG. 28B,a compliant shelf may include two brands of pasta products, products2804 and 2810, displayed on a top shelf 2812 labeled “pasta,” Shelvingunit 2802 may also display various canned goods, e.g., products 2814,2816, and 2818, on shelves 2820 and 2822, which both display a label for“canned goods.” This planogram, e.g., planogram 2808, may represent adesired appearance of shelving unit 2802, with product placementdetermined, for example, by market research entity 110 or supplier 115.

Discrepancy determination component 2704 may access the planograminformation, e.g., planogram 2808, and identify a discrepancy in theproduct placement when the desired placement of products indicates thatproducts from a first product type are to be placed on the at least oneshelf and the determined placement of products indicates that productsfrom a second product type are placed on the at least one shelf. Forexample, as shown in FIG. 28A, an identified discrepancy may be amisplaced box of pasta 2804 a on a shelf labelled “canned goods.”

In some embodiments, a discrepancy may be based on a product's placementrelative to a label, e.g., label 2806. For example, image analysiscomponent 2702 may identify a pasta box 2810 a on a bottom shelf of unit2802 and may also identify a shelf 2812 labelled “pasta.” Based on thedistance between the pasta box 2810 a and the “pasta” shelf label 2806being greater than an allowable threshold, discrepancy determinationcomponent 2704 may determine that the box 2810 a is misplaced, e.g., ison a non-compliant shelf. In another example, the discrepancy may bebased, at least in part, on a product's to at least one promotion sign.

Notice component 2706 may receive discrepancy information fromdiscrepancy determination component 2704, or based on other methods foridentifying discrepancies as described above, and may determine whethera notice-override condition exists relative to the identified at leastone product type due to misidentification of the product. For example,the system may typically notify an employee that a reorganization of ashelving unit should be performed if a particular product is misplaced.However, if the misplaced product is often misidentified, the system mayoverride the notice. For example, the “misplaced” product may have beenpreviously misidentified as a first product, when it is actually asecond product and is in the correct placement.

Notice-override conditions may be based on, for example, a brandassociated with the identified product type, a product size associatedwith the identified product type, an appearance of products of theidentified product type, the retail store (e.g., particular retailstore, the chain of the retail store, the type of the retail store, thegeographical location of the retail store, past events in the retailstore. etc.), the time the image was captured, the neighboring producttypes to the identified product type according to the at least oneplanogram, the neighboring product types to the identified product typeaccording to an actual placement of products, image capturing parameters(e.g., lower resolution images may tend to cause confusion),similarities of products of the identified product type to products ofat least one other product type (product type specified by the desiredplacement of products, product type of neighboring product, product typeidentified for products previously identified in the same location,etc.), past mismatch identifications between product types, and thelike. Additional parameters may be received by notice component 2706,for example, from database 140, market research entities 110, suppliers115, etc.

In some embodiments, system 100 may be configured to receive feedbackvia one or more user devices. For example, an employee may receive aninstruction to move a misplaced product. However, the employee maydetermine that the actual product was misidentified by the system and isin the correct place. The employee may input data indicative of themismatch, e.g., associating the correct product with the misidentifiedproduct. This information may be stored in a database, e.g., database140. Misidentified products may occur when one or more product typeshave a similar shape, coloring, brand, label, etc. For example, apackage of coffee may be misidentified as decaf due to a similar labelof the caffeinated coffee of the same brand. In another example,referring to FIG. 288, product 2814 and product 2818 may be oftenmistaken due to their similar size and shape.

The stored misidentification information may be used by notice component2706 to determine whether a notice-override condition exists for aproduct type identified by image analysis component 2702. For example,notice component 2706 may be configured to retrieve informationindicative of at least one past instance in which products from thefirst product type were erroneously identified as products from thesecond product type. Based on the at least one past instance, noticecomponent 2706 may determine existence of a notice-override conditionassociated with the identified at least one product type, wherein thenotice-override condition includes an erroneous identification of theproducts placed on the at least one shelf as products from the secondproduct type. In this case, notice component 2706 may withhold issuanceof a user-notification due to erroneous identification of the productsplaced on the at least one shelf as products from the second producttype.

In some embodiments, the image analysis component 2702 may also analyzethe image data to determine information about products placed adjacentto the products associated with the identified at least one producttype. In this case, the determination of whether a notice-overridecondition exists may be based on the information about the productsplaced adjacent to the products associated with the identified at leastone product type. For example, notice may be overridden if the visualappearance of neighboring products is similar.

In some embodiments, the system may determine patterns of productconfusion based on accessed information from one or more databases. Forexample, information in database 140 may include feedback from anemployee that the system issued a notice based on a misidentifiedproduct. In some embodiments, a confusion matrix of a likelihood thatone product type is mistaken for another product type may be constructedfrom past performances. As an example, a sequence of identifications ofa first product and a second product on a particular shelf may be(1,1,1,1,2,1,1,1,1,1,1,1,2,1,1). This may indicate that the firstproduct may tend to be mistakenly identified as the second product.Notice component 2706 may use the patterns of product confusion todetermine the existence of the notice-override condition. Thus, anotification alerting an employee to a product of the second producttype in the area of the first product may be overridden.

In cases in which a product is misidentified by the system, a notice maybe erroneously issued to a user. Upon issuance of a user-notificationassociated with the discrepancy between the planogram and the shelfimage, the system may receive data confirming that the discrepancyresulted from erroneous identification of the at least one product type.For example, an employee may input data indicative of a productmisidentification. The system may update the stored patterns of productconfusion to reflect that the discrepancy resulted from erroneousidentification of the at least one product type.

In another embodiment, image analysis component 2702 may receiveadditional image data captured at a second time after a determination towithhold issuance of the user-notification due to existence of thenotice-override condition. Discrepancy determination component 2704 mayanalyze the additional image data to determine that the discrepancyassociated with the at least one store shelf disappeared withoutintervention of a store employee. For example, the discrepancy may havebeen resolved without a restock or reorganization event. In this case,the system may update the patterns of product confusion to reflect thatthe discrepancy resulted from erroneous identification of the at leastone product type.

In another embodiment, notice component 2706 may determine whether anotice-override condition exists may be based on a last time that the atleast one store shelf was organized by a store employee. For example, ifthe shelf was recently restocked, the change in product placement mayhave been intentional, and thus, issuance of a user notification isunnecessary. In some embodiments, the information about a last time thatthe at least one store shelf was organized by a store employee is basedon an analysis of previous images captured from the retail store,information from a database of actions performed in the store, etc.

In some embodiments, a notice-override condition may be associated withan experience level of an employee. For example, notice component 2706may determine the last employee who worked on a particular shelf andaccess information, e.g., in database 140, associated with the employee.For example, the system may determine whether a notice-overridecondition exists based on a level of experience of the store employee.If an employee is new, it may be more likely that a product is misplacedand notice is required to instruct the employee to correct the placementof the product.

In some embodiments, the determination of whether a notice-overridecondition exists is based on information about past events of planogramincompliance associated with the at least one store shelf. For example,past events of planogram incompliance may indicate that the shelf is ina high-traffic area and customers often misplace products on the shelfIn another example, products associated with a promotion may be depletedat a faster rate, resulting in planogram incompliance. In someembodiments, details of past events may be received as feedback input byan employee and stored in database 140.

In some embodiments, the notice-override condition may be based oninventory information. For example, the determination of whether anotice-override condition exists is based on inventory informationassociated with the identified at least one product type. If a productis placed in the wrong area, but the store does not have inventory ofthe correct product, the system may determine not to send an alert to anemployee to restock the correct product. In another example, thedetermination of whether a notice-override condition exists is based onreference information associated with the identified at least oneproduct type from other retail stores. Thus, if the store cannottransfer more inventory from another location, a notice to restock thecorrect product may not be sent to an employee.

In some determining notice component 2706 may access patterns of productavailability of product types based on accessed information from one ormore databases. Notice component 2706 may use the patterns of productavailability to determine the existence of the notice-overridecondition. For example, no notice may be sent to restock a product ifthe availability pattern indicates that the product tends to run out atthe time of day at which the image of the shelving unit was captured.

In some embodiments, notice component 2706 may determine a certaintylevel for the notice-override condition and issue a user-notificationindicating that an event of planogram incompliance associated with theat least one store shelf had occurred when the certainty level is lessthan a threshold. The event of planogram incompliance may include aproduct being out of stock, an empty shelf, a shelf with incorrectproducts, etc.

The determination of the certainty level may be based on the at leastone identified product type. For example, the certainty level may belower for a product type that is misidentified at a particular rate(e.g., 25% of identifications of the product are incorrect, 10% ofidentifications are incorrect, etc.). The threshold for a certaintylevel may be based on the desired placement of products associated withthe at least one store shelf. For example, the threshold may be higherfor specific brands or products based on contractual obligations or rateof return on the product.

In some embodiments, image analysis component 2702 may receiveadditional image data captured at a second time after a determination towithhold issuance of the user-notification based on the existence of thenotice-override condition by notice component 2706. The system mayanalyze the additional image data to determine an updated certaintylevel for the notice-override condition. Notice component 2706 may issuea user-notification indicating that an event of planogram incomplianceassociated with the at least one store shelf had occurred when theupdated certainty level is less than a second threshold. In someembodiments, the notice may be issued with the second threshold is thesame as the first threshold. In other embodiments, notice may be issuedwhen the level of certainty is less than the first certainty level.

FIG. 29 is a flowchart of a method 2900 for processing images capturedin a retail store and automatically identifying situations to withholdplanogram incompliance notifications. Method 2900 may be implemented bya general-purpose computer or a special-purpose computer built accordingto embodiments of the present disclosure. For example, method 2900 maybe executed by at least one processor, e.g., processing unit 130.

At step 2902, the system, e.g., image analysis component 2702, mayreceive image data from a plurality of image sensors mounted in theretail store, wherein the image data depicts a plurality of productsdisplayed on at least one store shelf. For example, image sensors mayinclude capturing devices 125, which are mounted to a retail shelvingunit and configured to capture images of opposing shelving units.

At step 2904, image analysis component 2702 may analyze the image datato identify at least one product type associated with the plurality ofproducts displayed on at least one store shelf and to determine aplacement of products of the at least one product type on the at leastone store shelf. For example, as described above, image analysiscomponent 2702 may use one or more image analysis techniques torecognize one or more product types associated with products in thecaptured image.

At step 2906, discrepancy determination component 2704 may access atleast one planogram describing a desired placement of products of the atleast one product type on shelves of a retail store. For example,discrepancy determination component 2704 may query a database, e.g.,database 140, to retrieve one or more planograms associated with theproduct type or a combination of product types identified by imageanalysis component 2702.

At step 2908, discrepancy determination component 2704 may identify,based on the at least one planogram, a discrepancy between thedetermined placement of products associated with the identified at leastone product type and the desired placement of products associated withthe identified at least one product type.

At step 2910, notice component 2706 may determine whether anotice-override condition exists relative to the identified at least oneproduct type due to misidentification of the product. For example, thisdetermination may be made based on information associated with theproduct type, such as a pattern of misidentification. In otherembodiments, the determination may be made, in part, based on productavailability information or employee information.

At step 2912, if a notice-override condition is not determined to existrelative to the identified at least one product type, issue auser-notification associated with the identified discrepancy between thedetermined placement of products and the desired placement of products.A notification may include instructions for an employee to take aparticular action with respect to the discrepancy. For example, thenotification may instruct an employee to restock or reorganize a retailshelving unit.

At step 2914, if a notice-override condition exists relative to theidentified at least one product type, withhold issuance of auser-notification associated with the identified discrepancy between thedetermined placement of products and the desired placement of products.In some embodiments, the withholding of the user notification may bebased on a certainty level associated with the notice-overridecondition. For example, if the certainty level is above a threshold, auser notification may not be issued.

As described above, a plurality of capturing devices 125 may beconnected to various forms of power sources. For example, capturingdevices 125 may be configured to obtain electrical power from anassociated mobile power source (e.g., one or more batteries), anexternal power supply (e.g., a power grid), a wireless powertransmission system, and so forth. In some embodiments, capturingdevices 125 may be configured to dynamically obtain electrical powerfrom multiple sources. For example, a capturing device 125 may beconfigured to obtain electrical power from the power grid for a periodof time but may be configured to obtain electrical power from the mobilepower source at other times (e.g., when the camera is disconnected fromthe power grid. etc.). Accordingly, capturing device 125 (which mayinclude one or more housings, such as housings 502 and 504 as describedabove) may be configured to dynamically connect to one or more mobilepower sources. In some embodiments, the mobile power sources may beconfigured to attach to a shelving unit.

FIG. 30A is a schematic illustration of an example system 3000 foracquiring images of products in a retail store, consistent with thepresent disclosure. As shown, system 3000 may include a control unit3010, one or more mobile power sources 3020 (shown in FIG. 30A asmultiple mobile power sources 3020A and 3020B), and one or more imagecapture devices 3040. Similar to with system 500 (described above withrespect to FIG. 5A), in some embodiments, capturing device 125 may referto some or all of the components of system 3000 shown in FIG. 30A. Insome embodiments, one or more components of system 3000 may beconfigured to attach to a retail shelving unit 3001. For example, mobilepower source 3020 and control unit 3010 may be configured to attach to abottom of retail shelving unit 3001 via a track or other device. Anexample track system is described in greater detail below. Retailshelving unit 3001 may be a store shelf, as described in greater detailabove.

Control unit 3010 may be configured to control one or more image capturedevices 3040. In some embodiments, control unit 3010 may correspond tosecond housing 504 as described above. Accordingly, image capture device3040 may correspond to image capture device 506, as described in greaterdetail above. Control unit 3010 may include one or more components ofcapturing device 125, including those shown in FIG. 3. For example,control unit 3010 may include processing device 302, network interface306, memory interface 304, memory device 314, peripherals interface 308,etc. In sonic embodiments, image capture device 3040 may be integral tocontrol unit 3010, as shown in FIG. 3. In other embodiments, the controlunit may be included in a separate housing from image capture device3040, for example, as described above with respect to FIGS. 5A-7B.

As shown in FIG. 30A, control unit 3010 may connect with mobile powersource 3020. Mobile power source 3020 may include any device configuredto receive, store, and/or supply electrical power. Mobile power source3020 may supply power to control unit 3010, which may be used by controlunit 3010 for performing the various functions described throughout thepresent disclosure. Mobile power source 3020 may further be configuredto supply electrical power to image capture device 3040. In someembodiments, mobile power source 3020 may be configured to supply powerdirectly to image capture device 3040. In other embodiments, controlunit 3010 may be configured to convey some or all of the electricalenergy received from mobile power source 3020 to image capture device3040.

Mobile power source 3020 may further be configured to electricallyconnect to an electrical power grid 3050, as shown in FIG. 30A. An“electrical connection” as used herein may include any connection whereelectrical power and: or signals may be transferred between two or morecomponents. In some embodiments, this may be a physical connection,through which power and/or data may be transmitted. In some embodiments,this may include a wireless transmission of power and/or data. As usedherein, a “power grid” may refer to any interconnected network fordelivering power to one or more devices. In some embodiments, a powergrid may broadly refer to a municipal power grid, such as an electricaltransmission system for delivering power to a plurality of customers,which may include the retail store in which system 3000 is being used.In other embodiments, a power grid may refer to a more local powernetwork. Power grid 3050 may be a network of electrical connections forsupplying power to capturing devices 125 and/or other componentsassociated with system 100. For example, power grid 3050 may be local toa retail store, one or more aisles of shelves, one or more shelvingunits, etc. Accordingly, power grid 3050 may be configured to supplyelectrical power to multiple systems 3000. Mobile power source 3020 mayreceive power from power grid 3050, which it may convey to control unit3010 and/or image capture device 3040 and/or use to charge one or morebatteries included in mobile power source 3020 (described furtherbelow).

In some embodiments, mobile power source 3020 may be configured toelectrically connect to additional mobile power sources. For example,mobile power source 3020A may be configured to electrically connect toone or more additional mobile power sources 3020B. Each of mobile powersources 3020A and 3020B may be configured to provide electrical power tocontrol unit 3010 and/or image capture device 3040. In some embodiments,mobile power source 3020B may be configured to share energy with mobilepower source 3020A, or vice versa. For example, mobile power source3020B may be configured to transfer energy (either from one or morebatteries included in mobile power source 3020B, or from power grid3050) to mobile power source 3020A. Mobile power source 3020A may thentransfer the received energy to an external device (e.g., control unit3010 and/or image capture device 3040) or may store the received energy,for example, by charging a battery internal to mobile power source3020A.

In some embodiments, system 3000 may include additional devices, such asbeacon device 3060. Beacon device 3060 may be configured to emit one ormore wireless signals via a wireless protocol, such as BLUETOOTH™,BLUETOOTH LE™ (BLE), Wi-Fi, near field communications (NFC), or thelike. For example, the signal emitted by beacon device 3060 may beconfigured to enable external devices to calculate their location (e.g,,within the store, with respect to the beacon, using signals fromadditional beacons, and so forth). In embodiments where components ofsystem 3000 are affixed to a track system, beacon device 3060 may alsobe enabled to attach to and/or slide along the track. Beacon device 3060may be configured to receive power from power grid 3050 and/or frommobile power source 3020. Mobile power source 3020 may include variouselectrical connectors for electrically connecting mobile power source3020 to the various devices described herein. Such electrical connectorsare described in greater detail below with respect to FIGS. 31A and 31B.

FIG. 30B is an illustration of an example control unit 3010 and mobilepower sources 3020A and 3020B, consistent with the present disclosure.As described above, mobile power sources 2020 may be configured tosupply electrical energy to control unit 3010. For example, controluntil 3010 may be electrically connected to mobile power source 3020A asshown in FIG. 30B. Further, mobile power source 3020A may beelectrically connected to an additional mobile power source 3020B.Mobile power source 3020A may be configured to receive power fromelectrical grid 3050 and/or additional mobile power source 3020B.

Mobile power source 3020 may include a housing 3022, as shown in FIG.30B. Housing 3022 may be configured to contain various internalcomponents and circuitry of mobile power source 3020. Housing 3022 maybe formed from a variety of materials. In some embodiments housing 3022may be formed from a plastic material such as acrylic, acrylonitrilebutadiene styrene (ABS), high-density polyethylene (HDPE), low-densitypolyethylene (I,DPE), polyvinyl choloride (PVC), nylon, resin, etc.Housing 3022 may also be for need from other materials, such as metal,glass, wood, fiberglass, etc. In some embodiments, housing 3022 maycomprise more than one material, for example, with multiple layers orsubcomponents.

As described above, mobile power source 3020 may include one or morebatteries. Accordingly, housing 3022 may be configured to retain atleast one battery, which may include a rechargeable power source, or aone-time use power source. Housing 3022 may retain a single battery ormay be configured to retain a plurality of batteries. The at least onebattery may include any type of battery suitable for storing electricalpower. For example, the battery may include, but is not limited to, arechargeable lithium-ion battery, an alkaline battery, a nickel-metalhydride battery, a lead-acid battery, or the like.

Mobile power source 3020 may be configured to supply power to controlunit 3010 from a battery associated with housing 3022 for controllingone or more image capture devices, such as image capture device 3040. Insome embodiments, control unit may control multiple image capturedevices (e.g., two, three, four, or more image capture devices.). Insome embodiments, mobile power source 3020 may be configured to supplypower to control unit 3010 from the battery associated with housing 3022when power grid 3050 is unavailable. Power grid 3050 may be unavailable,for example, if power grid 3050 is offline or disabled (e.g., during apower outage, maintenance operation, an inadvertent disconnection,scheduled down time, etc.), if one or more components of system 3000 aredisconnected or moved, or various other scenarios where power grid 3050would not be able to supply power to mobile power source 3020.Accordingly, mobile power source 3020 may be configured to supply powerto control unit 3010 from a battery. Mobile power source 3020 may beconfigured to recharge the at least one battery when the power grid isavailable.

Further, as described above, mobile power source 3020 may be configuredto electrically connect to an additional mobile power source (e.g.,mobile power source 3020B). In some embodiments, mobile power sources3020A and 30:20B may be the same. For example, system 3000 may be amodular system in which any number of mobile power sources 3020 may beconnected together to supply power to control unit 3010 and/or otherdevices. Accordingly, mobile power source 3020A may have a length L1which may be equal to a length L2 associated with mobile power source3020B. In other embodiments, mobile power sources 3020A and 3020B may bedifferent. For example, mobile power source 3020B may be larger orsmaller than mobile power source 3020A. Accordingly, length L1 of thehousing of mobile power source 3020A may be greater than length L2 ofthe additional mobile power source 3020B.

In some embodiments, mobile power source 3020A may be connected tomobile power source 3020B to increase an energy storage capacity that isavailable to control unit 3010. For example, the capacity of the atleast one battery of mobile power source 3020A may be sufficient forcapturing a certain number of images (e.g., 5,000, 7,500, 10,000, orgreater). When mobile power source 3020A is connected to the additionalmobile power source, a combined capacity of their batteries maysufficient for capturing a greater number of images (e.g., at least10,000 images, at least 15,000 images, at least 20,000 images or more).In some embodiments, the capacities of mobile power source 3020A and30208 may be different. For example, in embodiments where mobile powersource 3020A has a length L1 that is greater than length L2 of mobilepower source 3020B, mobile power source 3020A may have a greater storagecapacity. While mobile power sources 3020A and 3020B are generally shownto be connected in series, in some embodiments, the mobile power sourcesmay be connected in parallel. For example, the control unit (e.g.,control unit 3010) may be configured to connect to a first mobile source(e.g., mobile power source 3020A) that is parallelly connected to asecond mobile source (e.g., mobile power source 3020B). The control unitmay be configured to draw power from the second mobile source beforedrawing power from first mobile source.

In some embodiments, the control unit may be configured to determineinformation pertaining to mobile power source 3020, which may be used togenerate notifications relating to mobile power source 3020. Forexample, a processor included in control unit 3010 (e.g., processingdevice 302) may be configured to determine that a battery associatedwith the second mobile power source is below a predetermined chargelevel. For example, the predetermined charge level may be a certainpercentage (e.g., 50%, 30%, 10%, etc.), an estimated time remaining(e.g., 1 day, 1 hour, 10 minutes, etc.), an estimated number of imagesremaining (e.g., 1,000 images, 500 images, 10 images, etc.) or the like.The processor may further be configured to, based on the determination,cause generation of a notification to a store employee for replacing thesecond mobile power source. For example, the second mobile power sourcemay comprise a rechargeable power source, and the control unit may beconfigured to notify a store employee when the usability of therechargeable power source at a selected time after being recharged isbelow a selected threshold. The notification may be presented to theemployee in various ways, for example through a speaker or other outputdevice associated with control unit 3010, through a light or othervisual display associated with control unit 3010, by transmitting anotification to an employee device (e.g., output device 145D),transmitting a notification to server 135, or any other means fornotifying an employee.

Similar to mobile power source 3020, control unit 3010 may include ahousing 3012. Control unit 3010 may further comprise a plurality ofconnectors, such as connectors 3014A, 3014B, and 3014C shown in FIG.30B. Each connector may be configured to enable a connection fortransferring power to and for receiving data from to a separate imagecapture device fixedly mounted to the retail shelving unit. For example,connector 3014A may be configured to transfer power and receive datafrom image capture device 3040, whereas connectors 3014B and 3014C mayeach transfer power to and receive data from separate image capturedevices.

In some embodiments, system 3000 may include an apparatus for slidablysecuring a selectable number of mobile power sources to a retailshelving unit. For example, control unit 3010 and mobile power source3020 may be configured to slidably engage with a track 3030, as shown inFIG. 30B. Track 3030 may be attached to a retail shelf such that mobilepower source 3020 and control unit 3010 may be slidably engaged withtrack 3030 to slide along the shelving unit. Track 3030 may include oneor more rail arms configured to hold mobile power source 3020 and/orcontrol until 3010. These rail arms are described in greater detailbelow with respect to FIG. 31A.

FIG. 30C illustrates an example retail shelving unit with a mobile powersources 3020 and a control unit 3010 slidably engaged along a track3030, consistent with the present disclosure. In some embodiments, track3030 may be integral to retail shelving unit 3001. For example, the railarms of track 3030 may be formed as part of shelving unit 3001.Accordingly, mobile power sources 3020A and 3020B and control unit 3010may be configured to slidably engage with retail shelving unit 3001. Inother embodiments, track 3030 may be separate from retail shelving unit3001 and may be configured to attach to retail shelving unit 3001. Forexample, a base portion (e.g., base 3032 described below with respect toFIG. 31A) may be mounted on a first surface of the retail shelving unitopposite to a second surface of the retail shelving unit designated forplacement of retail products, as shown in FIG. 30C. Track 3030 may bemounted to retail shelving unit 3001 by various methods, for example,using an adhesive (e.g., tape, glue, epoxy, etc.), staples, clips, pins,bolts, screws, nails, rivets, etc., or through manufacturing techniques(e.g., welding, creating from a mold, etc.). In some embodiments, thetrack and/or the retail shelving unit may have features such as notches,grooves, or cutouts to facilitate attachment.

FIG. 31A is a side view illustrating an example apparatus for slidablysecuring a selectable number of mobile power sources to a retailshelving unit, consistent with the present disclosure. As describedabove, system 3000 may include an apparatus, such as track 3030 forslidably mounting mobile power source 3020 and/or control unit 3010 to aretail shelving unit. Track 3030 may include a base 3032 configured tobe mounted on a first surface of the retail shelving unit. The firstsurface may be opposite to a second surface of the retail shelving unitdesignated for placement of retail products, as illustrated in FIGS. 30Aand 30C. In some embodiments, base 3032 may include an adhesive layer3033 for mounting the apparatus to the first surface of the retailshelving unit. Adhesive layer 3033 may include any substance suitablefor attaching the apparatus to the shelving unit, including, forexample, tape, glue, epoxy, or the like. In some embodiments, base 3032may be affixed to the first surface of the retail shelving unit by othermeans, including, for example, staples, clips, pins, bolts, screws,nails, rivets, hook-and-loop fasteners (e.g,, Velcro™, 3M Dual Lock™,etc.), suction cups, or any suitable fastening techniques. In someembodiments, base 3032 may be an integral part of the retail shelvingunit. For example, base 3032 may be formed integrally with shelving unitthrough manufacturing techniques such as welding, creating from a mold,stamping, or other forming processes.

The apparatus may further include one or more rail arms for holding themobile power source or the control unit. For example, track 3030 mayinclude a first rail arm 3034A extending from base 3032, and a secondrail arm 3034B extending from base 3032. First rail arm 3034A mayinclude a first rail 3036A protruding from the first rail arm andextending in a longitudinal direction along first rail arm 3034A.Similarly, second rail arm 3034B may include a second rail 3036Bprotruding from the second rail arm in a direction toward first rail arm3034A and extending in a longitudinal direction along second rail arm3034B. The first and second rails may be configured to engage withcorresponding grooves of one or more power sources to slidably securethe one or more power sources to the retail shelving unit. For example,mobile power source 3020 may include housing 3022, as described above,which may include a first guiding channel 3110A located on a first sideof the housing and a second guiding channel 3110B located on a secondside of the housing opposite to the first side of the housing. The firstand second guiding channels being configured to engage with a trackassociated with a retail shelving unit in order to secure the housing tothe retail shelving unit. For example, first guiding channel 3110A maybe configured to engage with first rail 3036A and second guiding channel3110B may be configured to be engaged with second rail 3036B.

The rail arms may be spaced such that mobile power source 3020 may beslidably engaged with track 3030. For example, the two opposingprotrusions of rail arms 3034A and 303413 may be separated by a distanceD, as shown in FIG. 31A. Distance D may be any suitable distance forengaging mobile power source 3020. In some embodiments, distance D maybe between 45 mm to 85 mm. Mobile power source 3020 may have a width Wand a thickness T. Width W may be any width suitable for engaging withtrack 3030. In some embodiments, the width W may be between 40 mm to 80mm and thickness T may be between 10 mm to 25 mm. Any other suitableranges of thicknesses and/or widths may also be used, for example,depending on the configuration of retail shelving unit 3001.

In some embodiments, the apparatus (e.g., track 3030) may be configuredsupport two or more mobile power sources (e.g., mobile power sources3020A and 3020B) configured to power a plurality of image capturedevices mounted on a side of the retail shelving unit facing anotherretail shelving unit. In some embodiments, the mobile power sources maybe located in separate housings, as shown in FIG. 30B. Track 3030 mayfurther be configured to direct a first connector of a first mobilepower source to a second connector of a second mobile power source,thereby allowing the first mobile power source to mate with the secondmobile power source, which may be slidably positioned adjacent the firstmobile power source along the track. For example, mobile power source3020 may include an electrical connector 3120 configured to connect to asecond mobile power source. Example connectors are described in greaterdetail below with respect to FIGS. 31B and 31C.

Track 3030 may be constructed of any suitable material for slidablyengaging one or more mobile power sources. For example, track 3030 maybe made, at least in part, of a plastic such as acrylic, acrylonitrilebutadiene styrene (ABS), high-density polyethylene (HDPE), low-densitypolyethylene (LDPE), polyvinyl choloride (PVC), nylon, resin, etc. Inother embodiments, track 3030 may be made, at least in part, of a metal,such as steel, aluminum, brass, copper, or any other suitable metal ormetal alloy. In some embodiments, the material of track 3030 may beflexible such that mobile power source 3020 and/or control unit 3010 maybe snapped into track 3030. In some embodiments, the mobile power sourcemay only be snapped into track 3030 but may require a tool for removal.This may prevent theft or tampering with the components of system 3000.Accordingly, the apparatus may include a locking mechanism (not shown inFIG. 31A) that allows for toolless engagement of a power source with therails, but may need a tool to disengage the power source from the rails.In some embodiments, the tool may be a prying device configured todeflect the first rail arm from and/or the second rail arm to allow themobile power source to be removed. In other embodiments, the tool mayallow the mobile power source to slide out of track 3030 on one end. Forexample, the tool may disengage a lock, a screw, or another memberlocated on one end of track 3030, whereby disengaging the member allowsthe mobile power source to slide out of the end of the track.

While track 3030 is shown in FIG. 31A and described above in connectionwith attachment to mobile power source 3020, control unit 3010 and otherdevices (e,g., beacon device 3060) may similarly connect to track 3030.For example, housing 3012 of control unit 3010 may include a firstguiding channel located on a first side of the housing and a secondguiding channel located on a second side of the housing opposite to thefirst side of the housing, similar to guiding channels 3110A and 3110B,as described above. The first and second guiding channels may beconfigured to slidably engage with a track associated with a retailshelving unit, such as track 3030. Accordingly, any of the features,dimensions, or other disclosures above with respect to mobile powersource 3020 may also apply to control unit 3010.

FIGS. 31B and 31C show example electrical connectors for connectingmobile power source 3020 to control unit 3010 and/or other mobile powersources, consistent with the present disclosure. In some embodiments,the various components may be configured to connect directly to eachother. FIG. 3113 shows an example mobile power source 3020 configured toconnect directly to control unit 3010. Mobile power source 3020 mayinclude a first electrical connector 3120A associated with housing 3022.First connector 3120A may be configured to electrically connect mobilepower source 3020 to a control unit for at least one image capturedevice, such as control unit 3010. Mobile power source 3020 may alsoinclude a second electrical connector 3120B associated with housing3022. Second electrical connector 3120B may be configured toelectrically connect mobile power source 3020 to an additional mobilepower source and/or a power grid. For example, mobile power source 3020Aand may be configured to connect to additional mobile power source 3020Bor may be configured to connect to power grid 3050. Accordingly, powergrid 3050 may have a connector that is interchangeable with electricalconnector 3120A such that it may connect to electrical connector 3120B.In some embodiments, the first electrical connector may be associatedwith, a first end of the housing, and the second electrical connectorconfigured to be electrically may be associated with a second end of thehousing opposite to the first end of the housing, as shown in FIG. 31B.Mobile power source 3020 may further include circuitry (not shown inFIG. 31B) configured to convey power from the additional mobile powersource to the at least one image capture device when the mobile powersource is connected to the additional mobile power source, and to conveypower from the power grid to the at least one image capture device whenthe mobile power source is connected to the power grid.

Electrical connectors 3120A and/or 3120B may be any suitable connectorsfor conveying electrical power and/or data between mobile power source3020 and control unit 3010 or between two mobile power sources 3020. Insome embodiments, electrical connectors 3120A and/or 3120B may be a USBconnector (e.g., USB-A, USB-B, USB-C, Lightning, Micro-A, Micro-B,Mini-A, Mini-B, 5-pin, etc.). Electrical connectors 3120A and/or 3120Bmay be various other forms of connectors, including DVI, DisplayPort,eSATA, Firewire, 30-pin, HDML, etc. In some embodiments, various otherforms of connectors may be used and the present disclosure is notlimited to the connectors described above. Accordingly, first electricalconnector 3120A may further be configured to transfer data betweenmobile power source 3020 and control unit 3010. For example, the datamay include data received from image capture device 3040, which may beconveyed through mobile power source 3020 and eventually throughcommunications network 150. In other embodiments, the data may includeinformation regarding mobile power source 3020, including a remainingcharge level, as described above.

Control unit 3010 may include a power port 3150 configured to enable anelectrical connection for receiving power from at least one mobile powersource (e.g., mobile power source 3020), as shown in FIG. 31B. In someembodiments, power port 3150 may be interchangeable with electricalconnector 3120B. Accordingly, electrical connector 3120A of mobile powersource 3020 may be configured to connect with either another mobilepower source 3020 or with control unit 3010. Thus, any number of mobilepower sources 3020 may be connected together and/or connected withcontrol unit 3010 as needed. In some embodiments, power port 3150 may beconfigured to connect directly to power grid 3050, without anintermediate mobile power source.

Control unit 3010 may further include a plurality of connectors 3014(e.g., connectors 3014A, 3014B, and 3014C as described above), each ofwhich may be configured to enable a connection for transferring power toand for receiving data from to a separate image capture device fixedlymounted to the retail shelving unit, as described above. The pluralityof connectors may be located on a first end of housing 3012 of controlunit 3010, and power port 3150 may be located on a second end of thehousing opposite to the first end of the housing, as shown in FIG. 31B.In some embodiments, electrical connectors 3014A, 3014B, and 3014C maycorrespond to data conduit 508, as described above. Accordingly, any ofthe descriptions or disclosures associated with data conduit 508 mayalso apply to plurality of electrical connectors 3014.

As shown in FIG. 31C, mobile power source 3020 and control unit 3010 maybe connected by a cable 3130. In some embodiments, cable 3130 may beintegral to at least one of mobile power source 3020 or control unit3010. In other embodiments, at least one end of cable 3130 may beremovable. For example, cable 3130 may have an electrical connectorconfigured to connect with power port 3150 and/or an electricalconnector configured to connect with electrical connector 3120A. Asimilar cable may be used to connect two mobile power sources 3020, toconnect mobile power source 3020 to power grid 3050, or to connect powergrid 3050 directly to control unit 3010. In some embodiments, system3000 may be configured such that mobile power source 3020 can connectwith control unit 3010 either directly or using cable 3130. For example,cable 3130 may have an electrical connector on a first end configured tointerface with electrical connector 3120A of mobile power source 3020(and/or an electrical connector of power grid 3050). Cable 3130 maysimilarly have an electrical connector on a second end of the cableconfigured to connect with power port 3150 of control unit 3010 or withelectrical connector 3120B of mobile power source 3020. Accordingly, thecomponents of system 3000 may connect either directly with each other orvia cable 3130, as needed. For example, cable 3130 may be used as anextender when connecting two components that are spaced apart from oneanother.

In some embodiments, electrical connectors 3120A and 3120B and powerport 3150 may be integral to track 3030. For example, rails 3036A and3036B may include conductive elements such that power and/or data may betransferred to mobile power source 3020 and//or control unit 3010. Forexample, guiding channels 3110A and 3110B may include correspondingconductive elements such that power may be transmitted from power grid3050 through track 3030 to mobile power source 3020 and/or control unit3010. In other embodiments, electrical connectors 3120A and 3120B andpower port 3150 may be wireless. For example, mobile power source 3020may be configured to wirelessly transmit power and/or data to controlunit 3010. Various other forms of electrical connectors may also beused.

FIG. 32 provides a flowchart representing an exemplary method 3200 forusing a dynamic number of power sources for a camera mountable on aretail shelving unit, in accordance with the present disclosure. Forpurposes of illustration, in the following description, reference ismade to certain components of system 3000 as deployed in theconfiguration depicted in FIGS. 30A and 31A. It will be appreciated,however, that other implementations are possible and that otherconfigurations may be utilized to implement the exemplary method. Itwill also be readily appreciated that the illustrated method can bealtered to modify the, order of steps., delete steps, or further includeadditional steps.

At step 3202, method 3200 may include fixedly mounting on a retailshelving unit at least one image capture device configured to captureimages of an opposing retail shelving unit. For example, image capturedevice 3040 may be fixedly mounted to retail shelving unit. 3001, asshown in FIG. 30A. In some embodiments the at least one image capturedevice may correspond to first housing 502. Accordingly, fixedlymounting the at least one image capture device may comprise mountingfirst housing 502 on first retail shelving unit 602 on a side of storeshelf 622 facing second retail shelving unit 604, as described above.Various other configurations of image capture devices are describedabove with respect to method 700.

At step 3204, method 3200 may include fixedly mounting a trackcomprising a plurality of rails to the retail shelving unit. Forexample, step 3204 may include mounting track 3030 to shelving unit3001. As described above with respect to FIG. 31A, track 3030 maycomprise a base (e.g., base 3032) configured to be mounted on a firstsurface of the retail shelving unit opposite to a second surface of theretail shelving unit designated for placement of retail products. Track3030 may further comprise a first rail arm (e.g., first rail arm 3034A)extending from the base. The first rail arm may include a first rail(e.g., first rail 3036A) protruding from the first rail arm andextending in a longitudinal direction along the first rail arm. Track3030 may further comprise a second rail arm (e.g., second rail arm3034B) extending from the base, the second rail arm including a secondrail (e.g., second rail 3036B) protruding from the second rail arm in adirection toward the first rail arm and extending in a longitudinaldirection along the second rail arm. The first and second rails may beconfigured to engage with corresponding grooves of one or more mobilepower sources to slidably secure the one or more power sources to theretail shelving unit.

At step 3206, method 3200 may include slidably engaging at least onemobile power source with the plurality of rails. For example, mobilepower source 3020 may be slidably engaged with track 3030, as describedabove. In some embodiments, the mobile power source may comprise ahousing (e.g., housing 3022) configured to retain at least one battery.The housing may include a first guiding channel located on a first sideof the housing and a second guiding channel located on a second side ofthe housing opposite to the first side of the housing. The first andsecond guiding channels may be configured to engage with a trackassociated with a retail shelving unit (e.g., track 3030) in order tosecure the housing to the retail shelving unit.

In some embodiments, the mobile power source may further comprise afirst electrical connector associated with the housing. The firstconnector may be configured to electrically connect the mobile powersource to a control unit for at least one image capture device. Forexample, the first electrical connector may electrical connector 3120A,as described above. The mobile power source may further comprise asecond electrical connector (e.g., electrical connector 3120B associatedwith the housing. The second electrical connector may be configured toelectrically connect the mobile power source to at least one of anadditional mobile power source or a power grid. The mobile power sourcemay further comprise circuitry configured to convey power from theadditional mobile power source to the at least one image capture devicewhen the mobile power source is connected to the additional mobile powersource, and to convey power from the power grid to the at least oneimage capture device when the mobile power source is connected to thepower grid.

At step 3208, method 3200 may include slidably engaging at least onecontrol unit with the plurality of rails. In some embodiments, thecontrol unit may be configured to electrically connect with the at leastone mobile power source and the image capture device. For example,control unit 3010 may be slidably engaged with track 3030, as describedabove. The control unit may comprise a housing (e.g., housing 3012),which may include a first guiding channel located on a first side of thehousing and a second guiding channel located on a second side of thehousing opposite to the first side of the housing. The first and secondguiding channels may be configured to slidably engage with a track(e.g., track 3030) associated with a retail shelving unit. In someembodiments, the control unit may further comprise a plurality ofconnectors (e.g., connectors 3014) located on a first end of thehousing. Each connector may be configured to enable a connection fortransferring power to and for receiving data From a separate imagecapture device fixedly mounted to the retail shelving unit. The controlunit may further comprise a power port (e.g,, power port 3150) on asecond end of the housing opposite to the first end of the housing. Thepower port may be configured to enable an electrical connection forreceiving power from at least one mobile power source.

As described above, a server may be configured to determine whether ashelf of a retail store is in compliance with a planogram based onimages captured from a plurality of capturing devices 125. The servermay further be configured to transmit alerts based on an identifiedincompliance event. For example, an alert may be sent to an employee ofthe retail store where the incompliance event occurred. In someembodiments, the identification of an incompliance event in one retailstore may affect actions taken by the server in response to incomplianceevents identified in other retail stores. For example, feedback receivedfrom an employee in response to an alert for an incompliance event in afirst store may indicate a proper remedial action to be performed inother stores. In another example, feedback may indicate that anincompliance event in another store was incorrectly identified.Accordingly, the action taken in response to an incompliance event maydepend on feedback received based on a previous incompliance event,which may allow for more accurate or relevant actions to be taken inresponse to the incompliance event.

FIG. 33 illustrates an example system 3300 for detecting planogramcompliance in multiple retail stores, consistent with the presentdisclosure. System 3300 may include a server 3335 configured to receiveand analyze image data captured in a plurality of retail stores 3305,such as retail stores 3305A, 3305B, and 33050. In some embodimentssystem 3300 may correspond to system 100, described in detail above. Forexample, server 3335 may correspond to server 135 as shown in FIG. 1.Accordingly, any of the above descriptions or embodiments pertaining toserver 135 may also apply to server 3335. Server 3335 may receive imagedata captured in retail stores 3305. For example, each of retail stores3305A-C may contain one or more capturing devices 125 as describedabove. Server 3335 may communicate with devices in retail stores 3305over a network connection, such as through communications network 150.Retail stores 3305A-C may correspond to retail stores 105A-C, asdescribed above in reference to FIG. I. Accordingly, any of the abovedescriptions or embodiments pertaining to retail stores 105A-C mayequally apply to retail stores 3305 While three retail stores 3305A-Care shown by way of example system 3330 may include more or fewer retailstores. Further, it should be understood that system 3300 is not limitedto the elements shown in FIG. 33 and may contain additional elements notshown, including any of the elements included in FIG. 1.

As described above, retail stores 3305 may include any an establishmentoffering products for sale by direct selection either physically orthrough virtual shopping. Retail stores 3305A-C may be an establishmentoperated by a single retailer (e.g., a supermarket) or an establishmentthat includes stores operated by multiple retailers (e.g., a shoppingmall). In some embodiments two or more of retail stores 3305 may berelated. In one example, as shown in FIG. 33, retail store 3305A may beof a same type as retail store 3305B. For example, retail stores 3305Aand 3305B may both be grocery stores. In some embodiments, retail stores3305A and 3305B may be owned by the same company or parent company. Forexample, retail stores 3305A and 3305B may be part of the same chain ofstores Accordingly, retail stores 3305A and 3305B may be governed by thesame or similar contractual planogram requirements. In otherembodiments, some or all of retail stores 3305 may be of differenttypes. For example, retail store 3305A may be a grocery store, whileretail store 3305C may be a big box store (e.g., selling housewaresand/or other general merchandise). In some embodiments, 3305C may havethe same or similar planogram requirements, especially with respect to aparticular product. Some or all of retail stores 3305 may have differentplanogram requirements, however. In some embodiments, retail stores mayinclude other types of stores than those depicted in FIG. 33, including,but not limited to, electronics stores, hardware stores, departmentstores, wholesale or discount clubs, clothing stores, conveniencestores, specialty stores (e.g., jewelers, florists, sporting goodsstores, etc.), or any other type of retail store.

Server 3335 may receive imams captured by capturing devices 125 includedin retail stores 3305 and may analyze the captured images to assessplanogram compliance by retail stores 3305. Server 3335 may determineappropriate actions to take in response to identified incomplianceevents associated with retail stores 3305 based on other incomplianceevents associated with retail stores 3305, in some embodiments, thedetermined response may be based on incompliance events occurring at adifferent store. For example, server 3335 may determine an appropriateaction in response to an incompliance event identified at retail store3305A based on an incompliance event identified at retail store 3305B or3305C. In some embodiments, the incompliance event may be a previousincompliance event from the same store. For example, historicalincompliance events from retail store 3305A may be assessed to determinean appropriate response to a current incompliance event.

FIG. 34 illustrates an example scenario for coordinating actions takenin response to identified incompliance events, consistent with thepresent disclosure. Server 3335 may be configured to receive image datafrom stores 3405A and 34058. Retail stores 3405A and 34058 maycorrespond to retail stores 3305, described above. Accordingly, retailstores 3405A and 3405B may be the same type of retail store or may be ofdifferent types. Server 3335 may receive image data associated withimages captured by capturing devices 3425A and 34258, which maycorrespond to capturing devices 125 as described above.

Server 3335 may analyze image data received from an environment ofretail store 3405 to determine that incompliance event 3420A hasoccurred. Incompliance event 3420A may indicate that shelf 3410A is notin compliance with one or more planograms associated with retail shelf3410A. For example, a planogram may dictate a certain placementrequirement for product 3412A, which may refer to a product of aparticular type. As used herein, a product type may refer to aparticular category of product (e.g., soda, beer, sports drinks, etc.),a particular brand (e.g., Coca-Cola™, Pepsi™, etc.), a particular modelor subtype (e.g., Diet Pepsi™, Crystal Pepsi™, etc.), a group ofindividual products (e.g., products having a certain expiration date,products associated with a particular shipment, etc,), or any otherclassification of product. The planogram may be associated withcontractual obligations and/or other preferences related to retailer,supplier, or manufacturer methodology for placement of products onretail store shelves. In response to incompliance event 3420A, server3335 may generate a notification 3442A and provide it to an employee3430A of store 3405A. Employee 3430A may be a human employee, a robot, astore owner, a manager, or the like. In some embodiments, notification3442A may be transmitted for display on an employee device 3440A ofstore employee 3430A, which may correspond to output device 145, asdescribed above.

Incompliance event 3420A may be any detected event in which shelf 3410Adoes not comply with an expected or desired condition. Incomplianceevent 3420A may correspond to the various examples of planogramcompliance described throughout the present disclosure. For example,incompliance event 3420A may be based on product facing, productplacement, planogram compatibility, price correlation, promotionexecution, product homogeneity, restocking rate, and planogramcompliance of adjacent products. Various examples of incompliance events3420 are provided below.

Notification 3442A may be any form of electronic communicationindicating that incompliance event 3420A has been detected. Notification3442A may be presented in various manners, including as an email, an SMSmessage, a MMS message, a app notification, an audible notification, avisual notification (e.g., an LED light, etc.), an in-app indicator, orthe like. In some embodiments, notification 3442A may be presented on aGUI, such as GUI 1120, 1130, and/or 1140, as described above. In someembodiments, notification 3442A may further include instructions foremployee 3430A to take an action in response to incompliance event3420A.

Server 3335 may further receive feedback 3444A in response tonotification 3442A. In this context, “feedback” may include any receivedinformation pertaining to incompliance event. In some embodiments,feedback 3444A may refer to a response from employee 3430A. For example,employee 3430A may input information pertaining to incompliance event3420A in employee device 3440A. In other embodiments, feedback 3444A maybe generated by an automated system (e.g., by a computer system ofretail store 3405A, a computer system ° la manufacturer or supplier ofproduct 3412A, etc.), by a manager of retail store 3405A, by a supplieror manufacturer (e.g., of product 3412A), or any other suitable sourceof feedback relating to an incompliance event. In some embodiments,feedback 3444A may indicate an action that was taken in response tonotification 3442A. In other embodiments, feedback 3444A may be aconfirmation of incompliance event 3420A. For example, employee 3430Amay confirm that incompliance event 3420 occurred. In some embodiments,feedback 3444A may dismiss or justify incompliance event 3420A. Forexample, feedback 3444A may indicate an exception to the planogramcompliance (e.g., based on a holiday, a special promotion, an expirationof a planogram or contractual obligation, a change in planogram orcontractual obligations, etc.), a reason why the planogram may not becomplied with (e.g., a product shortage, etc.), an indication that theincompliance event was detected by mistake, or the like. Various otherexamples of feedback are provided below. In some embodiments, feedback3444A may be based on an analysis of one or more additional imagescaptured after notification 3442A was provided. For example, the one ormore additional images may be analyzed to determine whether an actionwas taken in response to notification 3442A (e.g., within a particulartime frame), and/or to determine the type of action taken, and feedback3444A may include and/or be based on such determination. For example, amachine learning model may be trained using training examples todetermine whether actions were taken and/or the types of taken actionsfrom images and/or videos, and the trained machine learning model may beused to analyze the one or more additional images to determine whetherthe action was taken in response to notification 3442A, and/or todetermine the type of action taken. An example of such training examplemay include an image and/or a video, together with a label indicatingwhether an action was taken (e.g., before the image and/or the video wascaptured, while the image and/or video were captured, etc.) and/or thetype of action.

As described above, an incompliance event may be identified based onvarious scenarios. In some embodiments, incompliance event 3420A may bebased on an amount of product 3412A displayed on shelf 3410A. Forexample, an amount of product 3412A may fall below a selected threshold.The threshold may be selected based on a planogram, based on contractualobligations of retail store 3405A, based on historic amounts of product3412A in the retail store, based on amount of product 3412A in otherretail stores, based on area of shelf 3410A dedicated to product 3412A,and so forth. Some examples of such an amount may include number ofproduct facings (e.g., a number of product 3412A placed on the frontedge of shelf 3410A), an amount of inventory on shelf, a weight ofproducts, a volume of products, and so forth.

Accordingly, notification 3442A may include instructions for restockingat least part of shelf 3410A previously dedicated to product 3412A withproducts of a different product type. In some embodiments, feedback3444A may comprise an indication of a shortage of a product of aparticular type. For example, employee 3430A may indicate that product3412A is out of stock, that a shipment of product 3412A was notreceived, or the like. In some examples, feedback 3444A may comprise anindication of a repositioning of a product type to a different part ofretail store 3405A. In some examples, feedback 3444A may comprise anindication of a removal of a product of a particular type from a retailstore. For example, employee 3430A may indicate that retail store 3405Ano longer carries product 3412A, that product 3412A has beendiscontinued, etc. In some examples, feedback 3444A may comprise anindication of a reduction of a shelf area dedicated to product 3412A inretail store 3405A. In some examples, the feedback may comprise anindication of a lessening of an amount requirement related to product3412A. For example, retail store 3405A, or a supplier or manufacturer ofproduct 3412A may change a planogram associated with shelf 3410A tolessen the required amount of product 3412A.

In some embodiments, incompliance event 3420A may be based on a pricelabel related to product 3412A indicating an unexpected price for theparticular product type. In some examples, the first action and/or thesecond action and/or the third action may include modifying the pricelabel. For example, a price displayed on shelf 3410A (which may bedetected based on various image analysis techniques described above) maynot correspond to an expected price for that product type. In suchembodiments, feedback 3444A may comprise an indication of a price changefor product 3412A,,for example based on a change in pricing set byretail store 3405A, a supplier or manufacturer of product 3412A, apromotion of product 3412A, or the like.

In some embodiments, incompliance event 3420A may be based on product3412A (or products of another product type) being in an unexpectedposition within the retail store. For example, product 3412A may bedetected in another part of retail store 3405A, or a product other thanproduct 3412A may be detected on shelf 3410A. Accordingly, notification3442A may include instructions for moving the products from theunexpected position. In some examples, the feedback may comprise anindication that the products are rightfully placed in the unexpectedposition. For example, the unexpected position of product 3412A may bein a promotion area within retail store 3405A, and feedback 3444A maycomprise an indication of a current promotion of product 3412A.

In some embodiments, incompliance event 3420A may be based on a failureto identify at least one product in the retail store. For example,product 3412A may not be detected by any capturing devices within store3405A. Accordingly, in some examples, notification 3442A may includeinstructions for capturing at least one image of product 3412A in higherresolution than the image data captured using the capturing, devices. Inother examples, notification 3442A may include instructions for scanninga barcode associated with the at least one product, which may confirm apresence of product 3412A within retail store 3405A. In some examples,the first action and/or the second action and/or the third action mayinclude identifying the at least one product. Feedback 3444A maycomprise at least one image of the at least one product in the higherresolution or other identifying information of the at least one product.

Server 3335 may further receive an indication of a second incomplianceevent 3420B associated with shelf 3410B in retail store 3405B. Similarto incompliance event 3420A, incompliance event 3420B may be determinedbased on analysis of image data captured by capturing device 3425Bwithin an environment of retail store 3405B. In some embodiments,incompliance event 3420B may be associated with a product 3412B. Server3335 may determine that shelf 3410B is associated with shelf 3410B. Forexample, in some embodiments, server 3335 may determine that products3412A and 3412B are of the same type, as described above. For example,products 3412A and 3412B may be of the same product category, the samebrand, the same model or subtype, the same product group, may be placednext to similar neighboring products, etc. In some embodiments, shelf3410A may be similar to shelf 3410B in other respects. For example,shelves 3410A and 3410B may be in a similar location within the store(e.g., similar aisle, similar position in relation to a store entrance,a checkout station, a backroom, etc.), may have a similar shelf height,a similar shelf size, or the like. In some examples, a machine learningmodel may be trained using training examples to determine whether twoshelves are associated from images and/or videos of the two shelves, andthe trained machine learning model may be used to analyze an imageand/or a video of shelf 3410B and an image and/or a video of shell 3410Bto determine whether shelf 3410B is associated with shelf 3410B. Anexample of such training example may include an image and/or a video ofa first shelf and an image and/or a video of a second shelf, togetherwith a label indicating whether the two shelves are associated. In someexamples, a machine learning model may be trained using trainingexamples to determine whether two shelves are associated from datarelated to the two shelves, and the trained machine learning model maybe used to analyze data related to shelf 3410B and data related to shelf3410B to determine whether shelf 3410B is associated with shelf 3410B.Some non-limiting examples of such data related to a shelf may includelocation of a shell within a retail store, products corresponding to ashelf (e.g., according to a planogram, according to ana analysis ofimages of the shelf, etc.), height of shelf, structure of shelf,information related to nearby items in the retail store, and so forth.An example of such training example may include data related to a firstshelf and data related to a second shelf, together with a labelindicating whether the two shelves are associated.

Based on the determined association between shelves 3410A and 3410B,server 3335 may consider feedback 3444A in determining an appropriateaction to take in response. For example, in some embodiments, server3335 may generate a notification 3442B for an employee 3430B of store3405B, which may be presented on an employee device 3440B, similar tonotification 3442A. In some embodiments, notification 3442B may includeinstructions for employee 3430B, similar to the instructions describedabove with respect to notification 3442A. The content of notification3442B may be based, at least in part, on feedback 3444A. For example, iffeedback 3444A confirms the occurrence of incompliance event 3420A,notification 3442B may alert employee 3430B to incompliance event 3420B.Notification 3442B may include the same or different instructions asthose included in notification 3442A. In some embodiments, notification3442B may further include information about incompliance event 3420A,for example, a time of the event, a location of the event, a type ofevent, whether the incompliance was resolved, how the incompliance wasresolved, etc.

In another example, feedback 3444A may indicate an action that was takenin response to incompliance event 3420A. For example, feedback 3444A mayindicate how incompliance event 3420A was resolved. The action taken inresponse to incompliance event 3420A may or may not be the same as aninstructed action notification 3442A. For example, feedback 3444A mayinclude a confirmation that an instructed action was taken or mayindicate an alternative action. Notification 3442B may include aninstruction to take the same action that was taken in response toincompliance event 3420A. In other embodiments, the action taken inresponse to incompliance event 3420A may not resolve the incompliance.Accordingly, notification 3442B may include instructions to take adifferent action. In other embodiments, the instructions may be toconsult with employee 3430A, or with retail store 3405A for anappropriate action.

In another example, feedback 3444A may dismiss or otherwise resolveincompliance event 3420A. For example, feedback 3444A may indicate achange in planogram requirements, an exception to the planogramrequirements, extenuating circumstances (e.g., a product shortage, atemporary promotion or other change in planogram requirements, etc.), oranother reason why the incompliance is justified or not relevant. Thisinformation may be used in determining a course of action in response toincompliance event 3420B. For example, server 3335 may withholdnotification 3442B based on the information in feedback 3444A. If thesame information might be applicable to incompliance event 3420B. Inother embodiments, server 3335 may transmit notification 3442B but maywithhold instructions based on feedback 3444A. For example, notification3442B may alert employee 3430B as to the incompliance but may notinclude instructions. In some embodiments, notification 3442B mayinclude all or some of feedback 3444A. This may alert employee 3430B tothe conditions identified by employee 3430A, which may eliminate orreduce the investigation or other efforts required by employee 3430B. Insome embodiments, some or all of the information in feedback 3414A maybe provided in a notification to employee 3430B even before incomplianceevent 3420A is detected. Similarly, feedback 3444A may indicate thatincompliance event 3420A was detected erroneously. Accordingly, based oncontext provided in feedback 3444A, server 3335 may determine thatincompliance event 3420A was similarly erroneous and may foregogenerating notification 3442B.

In some embodiments, incompliance event 3420B may be detected prior toreceiving feedback 3444A. Server 3335 may forego action based onincompliance event 3420B at least temporarily, until feedback 3444A isreceived. In some embodiments, server 3335 may wait a predeterminedperiod of time (e.g., 5 minutes, 30 minutes, several hours, severaldays, several weeks, etc.) before taking action. Accordingly, server3335 may provide more relevant recommendations or otherwise take a moreappropriate action once feedback 3444A is received. If feedback 3444A isnot received before the predetermined period of time has elapsed,another action may be taken, for example, the same action taken inresponse to incompliance event 3420A.

FIG. 35 provides a flowchart representing an exemplary method 3500 forprocessing images captured in multiple retail stores and automaticallycoordinating actions across two or more retail stores, in accordancewith the present disclosure. For purposes of illustration, in thefollowing description, reference is made to certain components of system3300 as deployed in the configuration depicted in FIGS. 33 and 34. Itwill be appreciated, however, that other implementations are possibleand that other configurations may be utilized to implement the exemplarymethod. It will also be readily appreciated that the illustrated methodcan be altered to modify the order of steps, delete steps, or furtherinclude additional steps.

At step 3502, method 3500 may include receiving, first image datacaptured from an environment of a first retail store depicting productsdisplayed on shelves of the first retail store. For example, server 3335may receive image data captured by capturing device 3425A from within anenvironment of retail store 3405A. In some embodiments, the first imagedata may include images acquired by a plurality of image sensors fixedlymounted on at least one retail shelving unit of the first retail store,by image sensors mounted elsewhere in the retail store, by image sensorsof mobile devices, by robots configured to capture images, and so forth.Further details regarding mounted image sensors, such as capturingdevice 125, are provided above.

At step 3504, method 3500 may include analyzing the first image data toidentify a first planogram incompliance event associated with at leastone shelf of the first retail store, for example as described above. Forexample, the image data captured by capturing device 3425A may includeinformation identifying incompliance event 3420A associated with shelf3410A, as described above.

At step 3506, method 3500 may include generating an instruction for anemployee of the first retail store to perform a remedial action inresponse to the identified first planogram incompliance event. Forexample, a server, such as server 3335 may generate notification 3112A,which may include an instruction for employee 3430A to take a remedialaction in response to the identified incompliance event 3420A, asdescribed above. In some embodiments, the employee may be a human storeassociate. In other embodiments, the employee of the first retail storemay be a robot. In such embodiments, the feedback from the employee ofthe first retail store may be based on images (and/or on an analysis ofimages) captured by the robot.

At step 3508, method 3500 may include acquiring and storing feedbackfrom the employee or the first retail store regarding the firstplanogram incompliance event. The feedback may correspond to feedback3444A, as described above. The feedback may be stored, in memory, suchas in database 140, described above. In some embodiments, the employeemay input the feedback through a computing device, such as employeedevice 3440A. In some embodiments, the feedback may be detected based onactions of the employee detected by the capturing device. In otherwords, the feedback from the employee of the first retail store may beacquired from additional image data captured by the plurality of imagesensors fixedly mounted on the at least one retail shelving unit of thefirst retail store. For example, the feedback may include an indicationthat the employee has mitigated the incompliance event, which may bedetected based on the additional image data. In other embodiments, thefeedback may include an indication that the employee has determined notto take any action, or that an alternative action has been taken otherthan the instructed action.

At step 3510, method 3500 may include receiving second image datacaptured from an environment of a second retail store depicting productsdisplayed on shelves of the second retail store. For example, server3335 may receive second image data captured by capturing device 3425Bfrom within an environment of store 3405B. At step 3512, method 3500 mayinclude analyzing the second image data to identify a second planogramincompliance event associated with at least one shelf of the secondretail store. For example, server 3335 may identify incompliance event3420B associated with shelf 3410B, as described in greater detail above.

At step 3514, method 3500 may include determining that the at least oneshelf of the second retail store is related to at least one shelf of thefirst retail store. For example, server 3335 may determine that shelf3410A is related to shelf 3410B, as described in greater detail above.The at least one shelf of the second retail store may be related to atleast one shelf of the first retail store in various ways. For example,the at least one shelf of the first retail store and the second retailstore may hold products of a similar or same product type (e.g., samecategory, same brand, same model or subtype, etc.). In otherembodiments, the shelves may have a similar location within the store(e.g., in a similar aisle, in a similar position with respect to a storeentrance, checkout station, backroom, etc.). The shelves may sharevarious other properties, including similar shelf height, similar shelfsize, similar neighboring products, or any other common properties.

At step 3516, method 3500 may include accessing the feedback from theemployee of the first retail store regarding the first planogramincompliance event associated with the at least one shelf of the firstretail store. For example, server 3335 may access the feedback fromdatabase 140 or another storage location. At step 3518, method 3500 mayinclude using the feedback from the employee of the first retail storein determining what action to take in response to the second planogramincompliance event associated with at least one shelf of the secondretail store. The action may depend on the contents and the nature ofthe feedback, as described above. In some embodiments, step 3516 mayinclude, in response to a first feedback from the employee of the firstretail store, generating an instruction for an employee of the secondretail store to perform a remedial action in response to the identifiedsecond planogram incompliance event. Further, in response to a secondfeedback received from the employee of the first retail store, step 3516may include forgoing generating the instruction for the employee of thesecond retail store to perform the remedial action in response to theidentified second planogram incompliance event. For example, the firstfeedback may confirm the planogram incompliance event. Accordingly, theremedial action for the employee of the second retail store maycorrespond to the remedial action performed by the employee of the firstretail store in response to the identified first planogram incomplianceevent. In another example, the feedback may indicate an alternativecause for the identified planogram incompliance event, and that adifferent action should be instructed. Accordingly, the remedial actionfor the employee of the second retail store may be different from theremedial action performed by the employee of the first retail store inresponse to the identified first planogram incompliance event.

At step 3520, method 3500 may include initiating an action based on thedetermination. For example, initiating the action may includetransmitting a notification (e.g., notification 3442B), foregoingtransmitting a notification, transmitting a notification including aparticular instruction, transmitting a notification but omitting aninstruction, or various other actions. In some examples, the initiatedaction may include analyzing one or more additional images capturedafter a notification is provided to determine whether a recommended stepwas performed (for example, by a store employee, by a robot, throughcrowd sourcing, and so forth). For example, the recommended step mayinclude removal of an item (such as a promotion sign, a label, aproduct, etc.), and the one or more additional images may be analyzed todetermine whether the item was successfully removed or is still presentIn another example, the recommended step may include moving an item(such as a promotion sign, a label, a product, etc.) from a firstlocation to a second location, and the one or more additional images maybe analyzed to determine whether the item was successfully moved fromthe first location to the second location (i.e., to determine that theitem is not present at the first location and is present at the secondlocation) or not. In yet another example, the recommended step mayinclude restocking a retail shelf, and the one or more additional imagesmay be analyzed to determine whether the retail shelf was successfullyrestocked. In an additional example, the recommended step may includecleaning a particular part of a retail store (such as an aisle), and theone or more additional images may be analyzed to determine whether theparticular part is clean.

Method 3500 may be applied to various forms of planogram incomplianceevents, as described above. In some embodiments, identifying at leastone of the first planogram incompliance event and the second planogramincompliance event may include a detection of a number of products of aparticular product type falling below a predetermined threshold. Forexample, a planogram may require a predetermined number of products tobe displayed on a shelf and the incompliance event may include adetection of an insufficient number of displayed products. Accordingly,the remedial action in response to the identified first planogramincompliance event may include restocking at least part of a shelf withproducts of the particular product type. As another example, theremedial action may include restocking at least part of a shelfpreviously dedicated to the particular product type with products of adifferent product type. Further, the feedback acquired from the employeeof the first retail store regarding the first planogram incomplianceevent may include at least one of: an indication of a shortage ofproducts of the particular product type, an indication of arepositioning of the particular product type to a different part of thefirst retail store, an indication of a removal of the particular producttype from the first retail store, an indication of a reduction of ashelf area dedicated to the particular product type in the first retailstore, or an indication of a lessening of a requirement related to theparticular product type in the first retail store, an indication of amisidentification of the first planogram incompliance event, anindication of a change in a visual appearance of the particular producttype, an indication of a reason for an inability to take a firstremedial action (for example, a shortage of the particular product typein inventory, regulatory requirement, etc.), and so forth. Various otherexamples of employee feedback are provided above.

In some embodiments, identifying at least one of the first planogramincompliance event and the second planogram incompliance event mayinclude identifying a shelf label related to a particular product typeindicating an incorrect detail for the particular product type. Forexample, the label may display an incorrect wrong price, may advertise adate that has ended (e.g., a duration of a promotional event, etc.), mayindicate incorrect product information (e.g., incorrectly indicating aproduct is “Gluten free,” etc.), or any other information displayed on ashelf label that may not meet a planogram compliance requirement.Accordingly, the remedial action in response to the identified firstplanogram incompliance event includes modifying the label. For example,notification 3442A may include instructions to remove the label, replacethe label, update information on the label, etc. Consistent with thisexample, the feedback from the employee of the first retail storeregarding the first planogram incompliance event may include at leastone of: an indication of a price change for the particular product type,an indication of a promotion of the particular product type, and anindicator of a misplaced label, or any other information that may berelevant to the incompliance event. The action to take in response tothe second planogram incompliance event may be determined based on thisfeedback.

In some embodiments, identifying at least one of the first planogramincompliance event and the second planogram incompliance event mayinclude identifying products of a particular product type in anunexpected location within the retail store. Accordingly, the remedialaction in response to the identified first planogram incompliance eventmay include moving the products of the particular product type from theunexpected position. In some embodiments, the feedback from the employeeor the first retail store regarding the First planogram incomplianceevent may include an indication that the products of the particularproduct type are rightfully placed in the unexpected position. Forexample, the unexpected position may be a promotion area within thefirst retail store, and therefore the unexpected position may be valid.As another example, the feedback may include an indication of amisidentification of the type of the products. For example, the employeemay determine that the detected products are not actually of theparticular product type and thus the incompliance event is invalid.

In an embodiment., identifying at least one of the first planogramincompliance event and the second planogram incompliance event mayinclude a failure to identify at least one product in the retail store.For example, a planogram may require a specific product to be displayedon shelves of a retail store and server 3335 may not detect the specificproduct in the image data. Accordingly, the remedial action in responseto the identified first planogram incompliance event may include atleast one of: capturing at least one image of the at least one productin higher resolution than the first image data, scanning a barcodeassociated with the at least one product, or providing identifyingdetails on the at least one product. The first feedback may includeinformation associated with the remedial action (e.g,, higher resolutionimage, identifying details, etc.). Additional examples of incomplianceevents with associated instructions and feedback are provided in detailabove.

In some embodiments, the determined action may be based on whether theincompliance events are the same or similar. For example, method 3500may include determining whether a type of the first planogramincompliance event is the same as a type of the second planogramincompliance event. Method 3500 may further include, in response to adetermination that the type of the first planogram incompliance event isthe same as the type of the second planogram incompliance event,generating an instruction for an employee of the second retail store toperform a remedial action in response to the identified second planogramincompliance event. For example, if the first and second incomplianceevents are the same type, similar remedial actions may be required inthe first and second retail stores. Conversely, in response to adetermination that the type of the first planogram incompliance eventdiffers from the type of the second planogram incompliance event, method3500 may include forgoing generation of the instruction for the employeeof the second retail store to perform a remedial action in response tothe identified second planogram incompliance event. For example, thediffering types of incompliance events may indicate that an instructionprovided in response to the first incompliance event is not applicableto the second incompliance event.

As described above, image data associated v,Pith retail shelving may bereceived and analyzed by a server. In some embodiments, the server maybe configured to estimate shelf capacity based on the captured imagedata and provide suggestions for improving store procedures andreoccurring tasks. For example, the suggestions may help a retail storeoptimize product placement in shelves, optimize store or shelf layout,optimize inventory control (e.g., managing product storage, managingproduct orders, etc,), optimize employee tasks (e.g., shelf-restockingrates, etc.), meet customer needs, or otherwise improve store executionof procedures and tasks. In some instances, shelf capacity may not bereadily determined from the captured images and thus further processingand analysis may be undertaken. For example, the depth of the shelvingunit may be occluded by products placed on the shelving unit and/or theangle of the camera. As a result, a depth of the shelving unit may notbe geometrically measurable from the image, and the area of the portionof the shelving unit may be directly computable based on geometry.Instead, the depth and/or area and/or capacity of the shelving unit aremay be estimated from other visual cues, such as type of shelving unit,type of product, type of retail store, and so forth. Accordingly,capacity of products of a particular product type on a portion of ashelving unit may be estimated.

FIG. 36 illustrates an example environment 3600 for automaticallydetermining options for store execution based on shelf capacity,consistent with the present disclosure. Environment 3600 may be includedwithin a retail store, such as retail stores 105A-C, as described above.Environment 3600 may include a shelving unit 3610, which may be a retailstore shelf, as described in greater detail above. Shelving unit 3610may be configured to hold a plurality of products, as shown in FIG. 36.While shelving unit 3610 is illustrated in FIG. 36 as having a singleshelf for ease of illustration, it is understood that shelving unit 3610may comprise multiple shelves at various heights. Further, shelving unit3610 may have other forms. For example, at least a portion of shelvingunit 3610 may include a rack with products hanging on the rack. Otherforms of shelving unit 3610 may also be used in accordance with thedisclosed embodiments. Shelving unit 3610 may include a portion 3612dedicated to a product 3620 having a particular product type. Asdescribed above, a product type may refer to a particular category ofproduct(e.g., soda, beer, etc.), a particular brand (e.g., Pepsi®,etc.), a particular model or subtype (e.g., Diet Pepsi®, etc.), a groupof individual products (e.g., products having a certain expiration date,products associated with a particular shipment, etc.), or any otherclassification of product. In some embodiments, shelf portion 3612 maybe a certain portion of the width of a retail shelf, as shown in FIG.36. In other embodiments, shelf portion 3612 may include multipleshelves, a portion of multiple shelves, or any other section of shelvingunit 3610 that may be dedicated to a product of a particular type. Shelfportion 3612 may also be a rack, a bar, or other forms of displayingmultiple rows of items. Shelf portion 3612 may have a width W1 as shownin FIG. 36.

Images of shelving unit 3610 may be captured by capturing device 3625.In some embodiments, capturing device 3625 may be a handheld device, asshown in FIG. 36. For example, capturing device 3625 may be asmartphone, a tablet, a mobile station, a personal digital assistant, alaptop, a retail inventory device, or any other handheld device. Inother embodiments, capturing device 3625 may be fixedly mounted within aretail store (e.g., to store shelves, walls, ceilings, floors,refrigerators, checkout stations, displays, dispensers, rods which maybe connected to other objects in the retail store, and so forth) asdescribed above with respect to FIG. 4A. In some embodiments capturingdevice 3625 may correspond to capturing device 125, as described above.Accordingly, any of the above descriptions or embodiments pertaining tocapturing device 125 may also apply to capturing device 3625. As shownabove, capturing device 3625 may be configured to capture image dataassociated with shelving unit 3610. An example image 3626 that may becaptured by capturing device 3625 is shown on a display of capturingdevice 3625.

Image data captured by capturing device 3625 may be received at aserver, such as server 135 described above with respect to FIG. 1. Forexample, a set of images captured from retail stores 105A-C may betransmitted through communications network 150 and may be received atimage processing unit 130. Server 135 may be configured to receive a setof images and analyze the images to assess a shelf capacity and providesuggestions for improving store execution. This analysis is described ingreater detail below.

Shelving unit 3610 may have a depth that is occluded from view based onthe angle at which image 3626 is captured. For example, image 3626 showsa single row of eight products 3620 on shelf portion 3612. The actualproduct capacity for product 3620 on shelf portion 3612 may be greaterdue to the depth of shelving unit 3610. For example, as shown in FIG.36, shelf portion 3612 may have a product capacity 3630 that is based onmultiple rows of product 3620. Therefore, the depth of shelving unit3610 may not be geometrically measured from the image and the capacityof shelf portion 3612 may not be directly computed from image 3626.Accordingly, server 135 may be configured to analyze image 3626 and/orother images captured during a period time to determine a productcapacity for portion 3612 of shelving unit 3610.

To estimate a capacity for product 3620, server 135 may identify shelfportion 3612 based on captured image (or images) 3626. For example,using image recognition techniques described above, server 135 maydetect product 3620 within image 3626. Shelf portion 3612 may beidentified as the portion of shelving unit 3610 that is associated withproduct 3620 (e.g., the portion of shelving unit 3610 that is directlybelow instances of product 3620 within image 3626). In some embodiments,shelf portion 3612 may be identified using a trained machine learningmodel. For example, a machine learning model may be trained usingtraining examples to identify portions of shelving units dedicated toselected product types from images and/or videos. The trained machinelearning model may then be used to analyze the image data and identify aportion of the shelving unit dedicated to a particular product type. Anexample of such training example may include images and/or videosdepicting a shelving unit together with the desired identification ofportions of the depicted shelving unit dedicated to the selected producttypes. Another example of such training example may include imagesand/or videos depicting a shelving unit together with indication that nopart of the depicted shelving is dedicated to the selected producttypes. The machine learning algorithm may include a logistic regression,a linear regression, a regression, a random forest, a K-Nearest Neighbor(INN) model, a K-Means model, a decision tree, a cox proportionalhazards regression model, a Naïve Bayes model, a Support Vector Machines(SVM) model, an artificial neural network (such as convolutional neuralnetwork, deep neural networks, etc.), a gradient boosting algorithm, orany other form of machine learning model or algorithm.

To estimate the capacity of shelf portion 3612, server 135 may obtaindata indicative of a depth DI of the shelving unit. The depth data maybe obtained in various ways. In some embodiments, the depth of theshelving unit may be estimated based on the image data. For example, adepth of shelving unit 3610 may be estimated based on image 3626. Insome embodiments, a type of shelving unit dedicated to the particularproduct type may be used to estimate the depth of the shelving unit. Thetype of shelving unit may include a manufacturer or model of theshelving unit, a style of shelf of the shelving unit, or any otherclassification that may associated with a particular depth. In someexamples, a data structure connecting types of shelving with commondepths may be accessed to determine the corresponding depth. Forexample, a data structure may be stored in database 140 and accessed byserver 135. The type of shelf may be recognized by server 135 usingvarious image processing algorithms described above. For example,shelving unit 3610 may have distinguishing features represented in image3626 that may indicate a particular type of shelving unit. In someexamples, a machine learning model may be trained using trainingexamples to identify types of shelves from images and/or videos. Thetrained machine learning model may be used to analyze the captured imagedata and identify the type of the shelving unit dedicated to theparticular product type. An example of such training example may includeimages and/or videos depicting shelves (and in some cases an indicationof a selected portion of the shelves) together with their types. Thevarious machine learning models and/or artificial neural networksdescribed above may be used for determining a shelving unit type.

The data indicative of the depth of the shelving unit may be obtained invarious other ways. In some embodiments, the shelving depth (or ashelving type) may be estimated based on information about the retailstore. For example, a particular chain of retail stores may commonly beassociated with a particular shelving depth or shelving type, which maybe used to estimate depth. A data structure may be accessed associatingretail store types with particular shelving unit depths and/or types. Insome embodiments, the depth (or type) of the shelving unit may beprovided by an employee or other entity associated with the retailstore. In some embodiments, the particular type of product may be usedto estimate a depth of the shelving unit. For example, it may be commonin retail stores (or in retail stores of a particular type) to placesome product types on a particular type of shelf or on a shelf of aparticular depth. In some embodiments, a data structure may connectcertain product types with common shelf types or depths. The datastructure may be accessed by server 135 using the particular type ofproduct (e.g., product 3620) detected in the images to determine thedepth.

In some embodiments, a machine learning model may be used to estimatethe depth based on image data. For example, a machine learning model maybe trained using training examples to estimate depth of selectedportions of shelves from images and/or videos, and the trained machinelearning model may be used to analyze the image data and estimate thedepth of the portion of the shelving unit dedicated to the particularproduct type. An example of such training example may include imagesand/or videos depicting shelves (and in some cases an indication of aselected portion of the shelves) together with the desired estimation ofthe depth. Various other means for determining shelf depth may be usedconsistent with the present disclosure. In some embodiments, a machinelearning model may be used to estimate the product capacity for theportion of the shelving unit dedicated to the particular product typebased on image data. For example, a machine learning model may betrained using training examples to estimate product capacity of shelvesfrom images and/or videos, and the trained machine learning model may beused to analyze the image data and estimate the product capacity of theportion of the shelving unit dedicated to the particular product type.An example of such training example may include images and/or videosdepicting shelves (and in some cases an indication of a selected portionof the shelves) together with the desired estimation of the productcapacity.

Server 135 may then determine the product capacity for the portion ofthe shelving unit dedicated to the particular product type based on thedepth of the shelving unit. In some embodiments, the server may analyzethe set of images to determine at least one dimension of products fromthe particular product type. For example, the dimension may include awidth, height, or depth of the product. FIG. 37A illustrates exampledimensions for a product 3620 that may be determined for estimatingshelf capacity, consistent with the disclosed embodiments. As shown inFIG. 37A, product 3620 may have a width W2, a height H, and a depth D2.The dimensions for product 3620 may be determined in various ways. Someof the dimensions may be measurable based on received image data. Forexample, the height H and the width W2 of product 3620 may be measuredand/or estimated based on image 3626. In some embodiments, one or moredimensions may be determined based on the type of product 3620. Forexample, a type of product 3620 may be determined based on analysis ofthe images as described above. Dimensions associated with product 3620may be determined by accessing a data structure associating variousproduct types with known dimensions.

In some embodiments, server 135 may be configured to determine a productcapacity for the portion of the shelving unit dedicated to theparticular product type based on the depth of the shelving unit and theat least one dimension of products from the particular product type. Forexample, based on the dimension of products from the particular producttype determined above, server 135 may determine a maximum number ofproducts from the particular product type that can be placed in a frontrow of products associated with the portion of the shelving unitdedicated to the particular product type. The possible number ofproducts that may be placed on a front row can be obtained in a varietyof ways. In some embodiments, the possible number of front row productsmay be obtained from a user, for example through a user input. In someembodiments the number of front row products may be estimated byanalyzing image data. For example, a width W1 of shelf portion 3612(FIG. 36) may be compared to a product width W2 (FIG. 37A) to determinethe number of products having a width W2 that can fit within width W1.Server 135 may also account for spacing between products, which may bespecified by retailer and/or manufacturer preference (e.g., aplanogram), product properties, a default product spacing, or variousother values. In some embodiments, the number of front row products maybe estimated based on additional data or information. For example, thenumber of front row products may be estimated by analyzing readings frompressure sensors positioned on the shelf, estimated by analyzingreadings from light sensors positioned on the shelf, estimated based onthe particular product type, estimated based on measurements of theparticular type of product, estimated based on a type of the retailstore, or any other information accessible by server 135 that may berelevant. In some embodiments, a machine learning model may be used toestimate the number of front row products. For example, a machinelearning model may be trained using training examples to estimate thenumber of products that may be placed on a shelf for products ofselected product types in selected portions of shelves from imagesand/or videos. The trained machine learning model may be used to analyzethe image data and estimate the possible number of products of theparticular product type in the portion of the shelving unit dedicated tothe particular product type. An example of such training may includeusing images and/or videos depicting shelves with an indication of aselected portion of the shelves together with the desired estimation ofthe number of products that may be placed on the front row.

Further, sever 135 may be configured to determine, based on the obtainedshelf depth, a maximum number of products from the particular producttype that can be placed behind the front row of products. For example,server 135 may compare the estimated shelf depth D1 with a product depthD2 to determine the number of products of depth D2 that will fit withindepth D1. Server 135 may then determine a product capacity 3630associated with the portion of the shelving unit dedicated to theparticular product type based on the maximum number of products that canbe placed in the front row and the maximum number of products that canbe placed behind the front row. For example, the product capacity maycorrespond to the number of products that can be placed in the front rowtimes the maximum number of products of depth D2 that can fit withindepth D1.

Server 135 may be configured to generate at least one suggestion forimproving store execution using the determined product capacity. In someembodiments, the at least one suggestion may further be based on storeddata. Accordingly, server 135 may further be configured to access storeddata about a second plurality of products from the particular producttype. Each of the second plurality of products may be located separatelyfrom the shelving unit at the time the images were captured (e.g., whena period of time during which the set of images is captured ends).

FIG. 37B illustrates various stored data that may be accessed forgenerating a suggestion for improving store execution of procedures andtasks, consistent with the disclosed embodiments. The data may be storedon a database 3740, as shown. Database 3740 may include any memorydevice or other data storage location capable of storing data. In someembodiments, database 3740 may correspond to database 140. Accordingly,any of the example databases or other descriptions pertaining todatabase 140 may also apply to database 3740. Database 3740 may be partof server 135 or may be accessed by server 135 via a communication link.In some embodiments, the stored data may include a quantity of productsof the particular product type in a different location. For example,inventory data 3702 may indicate a number of product 3620 located instorage. In some embodiments, inventory data 3702 may refer to a numberof products stored in a storeroom or other storage area associated witha retail store. For example, inventory data 3702 may refer to aninventory of product 3620 included in a storage area of the retail storeincluding shelving unit 3610. In this context, a storage area may referto any location in which a plurality of products may be stored. Thestorage area may be a storage room, a portion of a shelving unit withinthe retail store dedicated to storage (e.g., a top shelf, etc.), or anyother suitable storage location. In other embodiments, inventory data3702 may refer to off-site storage, for example, in a warehouse, in adelivery truck, or the like.

In some embodiments, server 135 may access reference data 3704, whichmay include a quantity of products of the particular type available on ashelving unit of at least one other retail store. The other retail storemay be a retail store of the same type as the retail store associatedwith environment 3600. For example, the retail stores may be of the samechain, may be owned by the same parent category, may be of the samecategory (e.g., grocery store, hardware store, big box store, etc.), mayreceive shipments from the same supplier or suppliers, or may carryproducts of the same product type (e.g., product 3620). In someembodiments, server 135 may access stored sales data 3706, which mayinclude sales data associated with product 3620. For example, sales data3706 may indicate the number of products 3620 sold over a given timeperiod. The sales data may be local to the retail store includingenvironment 3600, may be associated with one or more other retailstores, or may be global or regional sales data associated with product3620. The sales data may be read from memory, may be received through acommunication network using a communication device, may be received froman external device, may be received from a sales database, and so forth.

Various forms of suggestions for improving the store execution may begenerated based on the product capacity of shelf portion 3612 and theaccessed data. In some embodiments, the suggestion may be intended toreduce a wasted demand for a product. As used herein, “wasted demand”may refer to instances where a customer wishing to buy a product of theparticular product type may have given up on buying the product becausethe customer failed to find the product in the retail store. Wasteddemand may be identified by comparing product capacity 3630 (and/orproduct population) to sales data 3706. In order to reduce wasted demandfor product 3620, server 135 may generate a suggestion to increaserestock rate of the particular product type, to increase number offacings of the particular product type, to increase shelf capacity ofthe particular product type, and so forth.

In some embodiments, the suggestion may include changing a size of theportion of the at least one store shelf dedicated to products from theparticular product type. For example, server 135 may determine that theproduct capacity for product 3620 does not coincide with the demand ofproduct 3620 (e.g., based on sales data 3706). In some embodiments,server 135 may generate a recommendation to increase or decrease widthW1 of shelf portion 3612. In another example, server 135 may recommendplacing product 3620 on a shelf with a larger or smaller capacity, on adifferent shelf type, etc.

In other embodiments, the suggestion for improving the store executionmay include changing a re-stocking frequency of products from theparticular product type at the at least one store shelf. For example,sales data 3706 may indicate that a product capacity of product 3620cannot meet a demand for product 3620 based on the current restockingrate. Accordingly, server 135 may generate a suggestion to increase are-stocking frequency of product 3620. Conversely, sales data 3706 mayindicate that the re-stocking frequency can be decreased based on salesdata 3706. In some embodiments, the recommendation to change re-stockingfrequency may be based on inventory data 3702. For example, server 135may determine that, based on the product capacity in shelf portion 3612and the inventory of product 3620, the re-stocking frequency can beincreased or decreased. This may ensure that a number of products on ashelving unit does not drop below a certain threshold (e.g., a thresholdset by a planogram, a minimum desired product availability, etc.).

In accordance with the present disclosure, the suggestion for improvingstore execution may include various shelf planning optimizationrecommendations. In some embodiments, the suggestion may includechanging a product assortment for the retail store or retail shelvingunit. For example, product capacity 3630 and/or other stored data mayindicate that too much (or too little) of a product type is displayed onshelving unit 3610 or in the retail store. In some embodiments, thesuggestion may include changing a position of products on the shelvingunit and/or changing space dedicated to products from the particularproduct type. For example, server 135 may recommend moving product 3620to a higher or lower shelf, moving product 3620 closer to an end ormiddle of an aisle, moving product 3620 to a different location in astore, etc. This may be based on additional data, such as sales data3706, which may indicate, for example, that the current position forproduct 3620 is not optimized based on a demand for product 3620. Insome embodiments, the suggestion may include recommended modificationsto shelving unit 3610, such as changing a type of shelf of the shelvingunit, changing a spacing between shelves of the shelving unit, or anyother modification to the shelving unit that may increase or decreasecapacity. For example, moving shelves vertically closer together mayallow additional shelving to be inserted, which may allow for greaterproduct capacity in order to meet demand, etc. Various other shelfplanning suggestions may be generated.

In some embodiments, the suggestion for improving the store executionmay include various inventory planning optimization recommendations. Forexample, the suggestion may be directed to optimizing product storage inthe storage area, managing a supply chain of a product type (e.g.,changing order rate, etc.), or the like. In some embodiments, thesuggestion may include changing a storage capacity of products from theparticular product type in the storage area. Similarly, the suggestionmay be an order recommendation for restocking the storage area. Forexample, storage capacity 3630 and/or sales data 3706 may indicate thata current inventory capacity and/or order frequency is insufficient tokeep up with the demand for the product. Accordingly, a suggestion toincrease the inventory and/or order frequency for the product may begenerated. In some embodiments, the suggestion may be for changing alocation of products from the particular product type in the storagearea. For example, server 135 may determine based on storage capacity3630, inventory data 3702, and/or sales data 3706 that inventory forproduct 3620 should be more readily accessible (e.g., located nearer toa entrance to the retail portion of the store, located nearer to aloading dock, located on a lower or higher shelf, etc.) in order toimprove efficiency of restocking or reordering product 3620.

In some example embodiments, the suggestion may be based at least inpart on a product population associated with product 3620. As usedherein, a product population may refer to an amount of products placedin a designated shelf portion compared to the capacity of the shelfportion. For example, the product population may refer to an estimatednumber of product 3620 located in shelf portion 3612 divided by thetotal capacity 3630 of shelf portion 3612. Accordingly, a productpopulation may be expressed as a percentage (e.g., 75%), a fraction(e.g., ¾, 0.75, etc.), a number of products (e.g., 30), or any othersuitable format. In some embodiments, the product population may bedetermined based on analyzing the set of images. For example, server 135may use various object detection or image processing techniques toestimate the number of products within shelf portion 3612 at the timethe image was taken. In some embodiments, this may be based on a machinelearning model. For example, a machine learning model may be trainedusing training examples (e.g., images and/or videos of shelving unitswith associated product populations) and may then be used to estimateproduct populations based on other images, such as image 3626. Theproduct population may also be determined from other information, suchas pressure sensor data, light sensor data, input by a store employee,or the like.

In some embodiments, server 135 may track product populations over aperiod of time (e.g., a period of time when the set of images wascaptured). Server 135 may identify correlations between the sales dataand the changes in levels of product population and may generate thesuggestion for improving store execution based on the correlation. Forexample, the correlation may indicate that sales are higher when theproduct population is higher (e.g., above a determined threshold, etc.).Accordingly, the suggestion to improve store execution may be toincrease a restock rate of product 3620, increase a product capacity ofshelf portion 3612, increase an inventory capacity of product 3620, orvarious other recommendations consistent with the present disclosure.

Some aspects of the present disclosure may involve comparing productpopulations to inventory data associated with a type of product 3620.For example, as described above, the stored data may include inventorydata 3702, which may be indicative or a number of a plurality ofproducts available in at least one storage area associated with theretail store. Server 135 may be configured to identifying changes in alevel of product population at shelf portion 3612 during the period oftime based on the set of images and the determined product capacity.Server 135 may then generate the at least one suggestion for improvingthe store execution based on the changes in the level of productpopulation and the inventory data. For example, the suggestion mayinclude increasing a restocking rate to ensure the product populationdoes not fall below a certain threshold, as descried above. In otherexamples, the suggestion may be to increase a product capacity, increasea size of shelf portion 3612, modify shelving unit 3610, increase anorder rate for product 3620, or the like.

In some embodiments, server 135 may generate a suggestion based onreference data indicative of a number of products available on ashelving unit of at least one other retail store, such as reference data3704, described above. For example, server 135 may identify changes in alevel of product population at shelf portion 3612 during a period oftime based on the set of images and the determined product capacity, asdescribed above. Server 135 may then generate and provide the suggestionfor improving the store execution based on the changes in a level ofproduct population and the reference data. For example. the referencedata may indicate that the same or similar product at the other retailstore is being depleted slower or faster than the product on shelvingunit 3610. The suggestion may be to match shelving unit 3610 with theshelving unit in the other retail store (e.g., modifying shelving unit3610, increasing or decreasing shelf portion 3612, or the like). Inother embodiments, the suggestion may be to differentiate shelving unit3610 from the other retail store in at least one aspect.

Similar to product populations, server 135 may track product facings fora particular product type. As used herein, product facings may refer tothe placement of a product in a particular orientation. In someembodiments, product facings refer to products along a front row or ashelf facing in a uniform orientation. Server 135 may compare thedetermined product capacity 3630 for the portion of the shelving unitdedicated to the particular product type with a capacity for productfacings for the particular product type at shelf portion 3612. Forexample, the capacity for product facings may represent a number ofproducts 3620 that may be included on a first row of shelf portion 3612.Server 135 may generate a suggestion for changing the capacity forproduct facings based on the comparison. In some embodiments, thesuggestion may depend on a ratio of the number of product facings to theproduct capacity. For example, when the product capacity is at least 5times the capacity of product facings, server 135 may suggest increasingthe capacity for product facings. This may include a suggestion forincreasing width of portion 3612, changing shelf spacing, or variousother actions. When the product capacity is below 5 times the capacityof product facings, another action may be suggested, such as foregoingproviding a suggestion, decreasing the capacity for product facings,increasing a restocking rate, or the like. While a ratio of 5 times thecapacity of product facings is used by way of example, various otherratios may be used.

In some embodiments, server 135 may detect a number of front facingproducts placed on the portion of shelving unit dedicated to theparticular product over a period of time. For example, server 135 mayanalyze a received set of images captured during a period of time andtrack a change in the number of front facing products over the period oftime. Using the product capacity and the detected changes in the numberof front facing products, server 135 may generate and display the atleast one suggestion for improving store execution. For example, thesuggestion may be to change a restocking rate of product 3620, increasea product facing capacity of shelf portion 3612, increasing an orderingrate for product 3620, or various other actions described herein.

In some embodiments, a planogram or contractual obligation may specify aminimum number of front facing products and the suggestion may begenerated to ensure the minimum number of front facing products is met.For example, server 135 may further use the determined product capacityand the detected changes in the number of front facing products topredict a time when the number of front facing products is less than athreshold. For example, a rate at which front facing products isdecreasing may be determined based on the number of front facingproducts detected in the images and timestamps associated with theimages. The rate may then be extrapolated to determine the predictedtime. In some embodiments, the rate may be compared to other data, suchas sales data 3706 which may be used to refine the time prediction. Forexample, the rate at which a number front facing products decreases maybe assumed to taper of at times when sales are expected to be lower(e.g., late at night, etc.). The threshold may correspond to a maximumnumber of product facings, a percentage of products with the desiredproduct facings (e.g., 75%, 50%, 30%, etc.), or any other suitablethreshold. In these embodiments, the suggestion may be for restockingthe portion of the shelving unit dedicated to the particular producttype before the predicted time. Accordingly, the suggestion may betransmitted to an employee (e.g., via output device 145C), as describedin greater detail below. In some embodiments, the suggestion may includecausing restocking shelf portion 3612 while the number of front facingproducts is at a maximum possible number of front facing productsassociated with shelf portion 3612. For example, server 135 may timerecommendations for restocking shelf portion 3612 to ensure that a frontrow of shelf portion 3612 stays full, or that each stack of productsalong a depth of shelf portion 3612 has at least one of product 3620,etc.

Consistent with the present disclosure, server 135 may be configured toassess one or more of the generated suggestions described above, in someembodiments, the suggestion may be assessed based on additional salesdata. For example, server 135 may obtain additional sales dataindicative of products 3620 sold during a second period of time afterthe at least one suggestion was implemented. For example, the secondtime period may be within a few days, a few weeks, a few months, etc.from when the suggestion was implemented. Server 135 may furtherdetermine whether implementing the at least one suggestion resulted inhigher sales of the particular product type during the second period oftime. For example, server 135 may access sales data 3706 to determinethe number (or rate) of sales of product 3620 during the second timeperiod as compared to the first time period (or another time periodbefore the suggestion being implemented). If the sales increased, server135 may correlate the increased sales to the suggested recommendation.Similarly, a decrease or consistency in sales data may indicate thesuggested recommendation was not successful or beneficial. Thecorrelation between the suggested recommendation and the effect on salesmay be used to improve future recommendations.

Server 135 may be configured to provide the at least one suggestion forimproving store execution. In some embodiments providing the at leastone suggestion may include transmitting a notification to a storeemployee. For example, the suggestion may be transmitted to an outputdevice (e,g., output device 145C) of a store employee, which may includea smartphone, a tablet, an augment reality headset, or the like. In someembodiments, the suggestion may be provided through a public device,such as a computerized display listing tasks in the retail store. Insome embodiments, the suggestion may be provided through a taskscheduling system (e.g., a supply chain management system, an employeetask management system, an employee work scheduling system, etc.), orany other system associated with the retail store or product. In someembodiments, providing the suggestion may include storing the suggestionin memory, such as in database 140 or database 3740. In someembodiments, the suggestion may be transmitted through communicationsnetwork 150 to another component of system 100. In some embodiments thesuggestion may correspond to notification 3442A or 3442B, as describedabove.

FIG. 38 provides a flowchart representing an exemplary method 3800 forprocessing images captured in a retail store and automaticallydetermining options for store execution based on shelf capacity,consistent with the disclosed embodiments. For purposes of illustration,in the following description, reference is made to certain components ofsystem 100 as deployed in the configuration depicted in FIGS. 1 and 36.It will be appreciated, however, that other implementations are possibleand that other configurations may be utilized to implement the exemplarymethod. It will also be readily appreciated that the illustrated methodcan be altered to modify the order of steps, delete steps, or furtherinclude additional steps,. In some embodiments, method 3800 may beperformed by at least one processing device, which may include server135.

At step 3802, method 3800 may include receiving a set of images capturedduring a period of time. For example, the images may be captured byimage capture device 3625, as described above. The set of images maydepict a first plurality of products from a particular product typedisplayed on a shelving unit in a retail store. For example, the imagesmay depict a first plurality or product 3620 included on shelving unit3610.

At step 3804, method 3800 may include analyzing the set of images toidentity a portion of the shelving unit dedicated to the particularproduct type. For example, the set of images may be analyzed to detectshelf portion 3612. As described above, the portion of the shelving unitdedicated to the particular product type may be determined using one ormore object detection or image recognition algorithms. In someembodiments, the portion of the shelving unit dedicated to theparticular product type may be determined using a trained machinelearning model.

At step 3806, method 3800 may include determining a product capacity forthe portion of the shelving unit dedicated to the particular producttype. For example, the product capacity may correspond to productcapacity 3630 described above. The determined product capacity may begreater than a number of the first plurality of products depicted in theset of images, as shown in FIG. 36. In some embodiments, step 3806 mayinclude obtaining data indicative of a depth of the shelving unit. Forexample, the depth may be obtained using image data to identify the typeof the shelving unit, which may be indicative of a depth of the shelvingunit. In another example, the depth may be estimate based on the type ofproduct placed on the shelf. For example, products of the particularproduct type may be commonly associated with a certain shelf depth.Similarly, the shelf depth may be estimated based on details about thestore. In another example, the shelf depth may be received from store aemployee, for example through a GUI of an employee device.

Step 3806 may further include determining the product capacity for theportion of the shelving unit dedicated to the particular product typebased on the depth of the shelving unit. In some embodiments, step 3806may include analyzing the set of images to determine at least onedimension of products from the particular product type. For example, theset of images may be analyzed to determine a depth D2 of product 3620using the various methods described above. Step 3806 may includedetermining the product capacity for the portion of the shelving unitdedicated to the particular product type based on the depth of theshelving unit and the at least one dimension of products from theparticular product type.

In some embodiments, step 3806 may include determining a maximum numberof products from the particular product type that can be placed in afront row of products associated with the portion of the shelving unitdedicated to the particular product type. For example, the maximumnumber of products that can be placed in the front row may be based onthe at least one dimension of products from the particular product type.Step 3806 may further include determining, based on the obtained shelfdepth, a maximum number of products from the particular product typethat can be placed behind the front row of products. The productcapacity associated with the portion of the shelving unit dedicated tothe particular product type may then be determined based on the maximumnumber of products that can be placed in the front row and the maximumnumber of products that can be placed behind the front row, as describedabove.

At step 3808, method 3800 may include accessing stored data about asecond plurality of products from the particular product type, whereineach of the second plurality of products is located separately from theshelving unit when the period of time ends. In some embodiments,accessing the stored data may include accessing a database, such asdatabase 3740, or another memory device or storage location. Variousexamples of stored data types are described above with respect to FIG.37A.

At step 3810, method 3800 may include using the determined productcapacity and the accessed data to generate at least one suggestion forimproving store execution. In some embodiments, the at least onesuggestion for improving the store execution may include changing a sizeof the portion of the at least one store shelf dedicated to productsfrom the particular product type, as described above. In someembodiments, the at least one suggestion for improving the storeexecution may include changing a re-stocking frequency of products fromthe particular product type at the at least one store shelf. The atleast one suggestion may further include a shelf planning optimizationaction, such as changing product assortment for the retail store,changing product assortment for the shelving unit, changing position ofproducts in the shelving unit, changing space dedicated to products fromthe particular product type, changing type of shelf of the shelvingunit, or changing spacing between shelves of the shelving unit. In someembodiments, the at least one suggestion may include an inventoryplanning optimization action, such as changing storage capacity ofproducts from the particular product type in the store's backroom,changing a location of products from the particular product type in thestore's backroom, or providing order recommendation for restocking thestore's backroom.

In some embodiments, the stored data may include sales data indicativeof a number of the second plurality of products sold during the periodof time and step 3810 may include identifying changes in a level ofproduct population during the period of time and determining the atleast one suggestion for improving the store execution based onidentified correlations between the sales data and the changes in thelevel of product population. In some embodiments, the stored data mayinclude inventory data indicative of a number of the second plurality ofproducts available in at least one storage area associated with theretail store. For example, the stored data may include inventory data3702, as described above. Step 3810 may include identifying changes in alevel of product population at the portion of the shelving unitdedicated to the particular product type during the period of time basedon the set of images and the determined product capacity; generating theat least one suggestion for improving the store execution based on thechanges in the level of product population and the inventory data; anddisplaying the at least one suggestion to an employee of the retailstore. Further, in some embodiments, the stored data may includereference data indicative of a number of the second plurality ofproducts available on a shelving unit of at least one other retailstore. For example, the stored data may include reference data 3704.Step 3810 may include identifying changes in a level of productpopulation at the portion of the shelving unit dedicated to theparticular product type during the period of time based on the set ofimages and the determined product capacity and generating and providingthe at least one suggestion for improving the store execution based onthe changes in a level of product population and the reference data.These processes are described in greater detail above.

In some embodiments, step 3810 may include comparing the determinedproduct capacity for the portion of the shelving unit dedicated to theparticular product type with a capacity for product facings for theparticular product type at the portion of the shelving unit dedicated tothe particular product type and providing the at least one suggestionbased on a result of the comparison. For example, step 3810 may include,in response to a first result of the comparison, providing a suggestionfor changing the capacity for product facings for the particular producttype at the portion of the shelving unit dedicated to the particularproduct type; and in response to a second result of the comparison,forgoing providing the suggestion.

In some embodiments, the at least one suggestion may be based at leastin part on product facings, as described above. For example, step 3810may include analyzing the set of images to detect changes during theperiod of time in a number of front facing products placed on theportion of the shelving unit dedicated to the particular product typeand using the product capacity and the detected changes in the number offront facing products to generate and display the at least onesuggestion for improving store execution. The at least one suggestionmay include changing a restocking schedule of products from theparticular product type to reduce a time duration where the number offront facing products is less than a threshold. In some embodiments,step 3810 may include using the determined product capacity and thedetected changes in the number of front facing products to predict atime when the number of front facing products is less than a thresholdand determining at least one suggestion for restocking the portion ofthe shelving unit dedicated to the particular product type before thepredicted time. In some embodiments, step 3810 may include causingrestocking of the portion of the shelving unit dedicated to theparticular product type while the number of front facing products is ata maximum possible number of front facing products associated with theportion of the shelving unit dedicated to the particular product type,as described above.

At step 3812, method 3800 may include providing the at least onesuggestion for improving store execution. For example, step 3812 mayinclude providing the at least one suggestion to art employee of theretail store, to a supplier of the retail store, to a robot, to anexternal store rearrangement service, to a store optimization module, orthe like.

In some embodiments, method 3800 may include additional steps not shownin FIG. 38. For example, method 3800 may include performing anassessment of the at least one suggestion, as described above. Method3800 may include obtaining additional sales data indicative of productsfront the particular product type sold during a second period of timeafter the at least one suggestion was implemented. Method 3800 mayfurther include determining whether implementing the at least onesuggestion resulted in higher sales of the particular product typeduring the second period of time.

Retailers may need to deal with products running out of stock on aregular or even daily basis. These out-of-stock events have a cumulativeeffect that may substantially and adversely affect the retailers'profit. One might think this problem may be solved easily by simplymonitoring the store shelves with a camera and an inventory managementsystem that triggers low-stock alerts when the number of products of aspecific product type reaches a predetermined threshold. But experienceteaches that this solution often fails to eliminate out-of-stock eventsfor several reasons. First, an image processing system may havedifficulty determining the exact number of products stocked on a shelfbecause the captured images typically depict only the first row ofproducts. This means that the camera may not have sufficient visibilityof the products in any rows behind the first row. Second, retail storesoften work under the premise that having a large number of productsfacing customers drives product sales. Accordingly, store employees aretypically instructed to organize products to create the look of aperfectly stocked shelf by maximizing the number of products display inthe first row. To obtain a more accurate understanding of when aparticular product's level of stock is waning, the present disclosuredescribes using Machine Learning (ML) algorithms or other predictivealgorithms to predict when a next low-stock event will occur andgenerate an alert associated with the predicted next low-stock event.

FIGS. 39A-39F illustrate six images captured in a retail store,consistent with the present disclosure. The six images represent achronological sequence of events illustrating inventory changes in atleast a portion of a retail shelving unit dedicated to products of aspecific product type. In the following example, the image illustratedin FIG. 39A was captured before the image of FIG. 39B, the imageillustrated in FIG. 39B was captured before the image of FIG. 39C, andso forth through FIG. 39E.

The image illustrated in FIG. 39A depicts a restocking event associatedwith a store shelf 3900. A restocking event may be determined when atleast a portion of a retail shelving unit is replenished with additionalproducts. As illustrated, a store employee 3902 (or other individuals)may replenish store shelf 3900 with additional products of the specificproduct type from cart 3904. The image illustrated in FIG. 39B depicts afirst low-facing event associated with a stare shelf 3900. A low-facingevent may be determined when the number of products placed in a frontrow of products in at least a portion of the retail shelving unit isJess than a facing threshold. For example, if the facing threshold is50%, a low-facing event may be determined when the number of productsplaced in a front row of products of store shelf 3900 is three. In oneembodiment, after identifying a low-facing event, the system may issue anotification that causes a store employee to reorganize the products onthe store shelf. The image illustrated in FIG. 39C depicts a firstfacing event associated with a store shelf 3900. A facing event may bedetermined when products placed on at least a portion of the retailshelving unit are reorganized by a store employee such that moreproducts are placed in a front row of at least a portion of the retailshelving unit. For example, store employee 3906 (or other individual)may reorganize the products on store shelf 3900 such that seven productsmay be placed in the front row of products of store shelf 3900. Theimage illustrated in FIG. 39D depicts a second low-facing eventassociated with store shelf 3900 similar to the first low-facing eventillustrated in FIG. 39B. The image illustrated in FIG. 39E depicts asecond facing event associated with store shelf 3900 similar to thefirst facing event illustrated in FIG. 39C. The image illustrated inFIG. 39F depicts a low-stock event associated with store shelf 3900. Alow-stock event may be determined when the number of products placed inat least a portion of the retail shelving unit is less than a quantitythreshold. For example, if the quantity threshold is 5%, a low-stockevent may be determined when the number of products placed in storeshelf 3900 is less than two.

FIG. 40A illustrates two graphs that schematically show changes ofproducts from a specific product type placed on a shelf of a retailshelving unit during two periods of time. The information presented inFIG. 40A may be determined from images that are continuously orperiodically received from at least one image capturing device thatmonitors the shelving unit. In one example, the upper graph shows theidentified changes in the facing percentage of products from thespecific product type. To determine the facing level, the system maydetermine a maximum number of products from the specific product typethat can be placed in a front row of products associated with theportion of the shelving unit dedicated to the specific product type andcalculate how much of the front row of products is populated. The uppergraph has an example Facing threshold that is set to 50%. When thefacing level reaches the facing threshold, the system may determine thata low-facing event is starting. In another example, the lower graphshows the estimated changes in the quantity percentage of products fromthe specific product type. The quantity level depicted in the lowergraph is estimated because the system typically cannot accuratelydetermine the quantity of the products on a shelf using only image data.The lower graph has an example quantity threshold that is set to 15%.When the estimated quantity of products from the specific product typeon the store shelf reaches the quantity threshold, the system maydetermine that a low-stock event is starting.

The first time period starts at t1 when the system detects a firstrestocking event during which a store employee replenishes a store shelfdedicated to products from a specific product type. Between t1 and t2,the facing level and the quantity of products from the specific producttype decrease, and at t2 the system detects a first facing event inwhich the facing level returns to 100%. For example, the system detectsa store employee has reorganized the products on the store shelf. Next,between t2 and t3, the facing level and the quantity of products fromthe specific product type decrease, and at t3 the system detects a firstlow-facing event. For example, the system may determine that the facinglevel in the associated store shelf is at 50% and assign a facing taskto a store employee to reorganize the products on the store shell.Between t3 and t4, the facing level and the quantity of products fromthe specific product type continue to decrease, and at t4 the systemdetects a second facing event during which the facing level returns to100%. For example, the second facing event may be triggered by thesystem after detecting the first low-facing event. Between t4 and t5,the facing level and the quantity of products from the specific producttype continue to decrease, and at t5 the system detects a secondlow-facing event. Between t5 and t6, the facing level and the quantityof products from the specific product type continue to decrease, and att6 the system detects a low-stock event. In one example, the system maydetermine that the quantity of products from the specific product typeplaced on the store shelf is at 15% and assign a restocking task to astore employee to replenish the store shelf with additional products.Between t6 and t7, the facing level and the quantity of products fromthe specific product type continue to decrease, and at t7 the systemdetects a second restocking event during which a store employeereplenishes the store shelf with products from the specific producttype. At t7, both the quantity and the facing level of products from thespecific product type return to 100%. In one example, the secondrestocking event may be triggered by the system after detecting thelow-stock event at t6. As discussed above, the system provides asolution to out-of-stock events in retail stores. With reference to theillustrated example, the system aims to minimize and preferably toeliminate the time duration between t6 and t7.

The second time period starts at t7 when the system detects the secondrestocking event. Between t7 and t8, the facing level and the quantityof products from the specific product type decrease, and at t8 thesystem detects a third low-facing event. Between t8 and t9, the facinglevel and the quantity of products from the specific product typecontinue to decrease, and at t9 the system detects a third facing eventduring which the facing level returns to 100%. Between t9 and t10, thefacing level and the quantity of products from the specific product typedecrease, and at t10 the system detects a fourth low-facing event.Between t10 and t11, the facing level and the quantity of products fromthe specific product type continue to decrease, and at t11 the systemdetects a fourth facing event during which the facing level returns to100%. Between t11 and t12, the facing level and the quantity of productsfrom the specific product type continue to decrease, and at t12 thesystem generates an alert associated with a predicted next low-stockevent. The alert may be generated before the quantity of the productsfrom the specific product type reaches to the quantity threshold. In oneembodiment, the system may time the issuance of the alert such that thenext restocking event will take place before the quantity of productsfrom the specific product type reaches the quantity threshold.Specifically, the system may take into account the time it would takefor a store employee to transport additional products from a storagearea and replenish the store shelf. Between t12 and t13, the facinglevel and the quantity of products from the specific product typecontinues to decrease, and at t13 the system detects a fifth low-facingevent. Between t13 and t14, the facing level and the quantity ofproducts from the specific product type continue to decrease, and at t14the system detects a third restocking event. As shown, the thirdrestocking event takes place before the quantity of products from thespecific product type reaches the quantity threshold.

FIG. 40B illustrates an exemplary embodiment of a memory device 4000containing software modules consistent with the present disclosure. Inparticular, as shown, memory device 4000 may include a retail storecommunication module 4002, a captured data analysis module 4004, apattern determination module 4006, a low-stock prediction module 4008,an alert generation module 4010, a database access module 4012, and adatabase 4014. Modules 4002, 4004, 4006, 4008, 4010, and 4012 maycontain software instructions for execution by at least one processor(e.g., processing device 202) associated with system 100. Retail storecommunication module 4002, captured data analysis module 4004, patterndetermination module 4006, low-stock prediction module 4008, alertgeneration module 4010, database access module 4012, and database 4014may cooperate to perform multiple operations. For example, retail storecommunication module 4002 may receive image data from one or more imagecapturing devices 125 in retail store 105. Captured data analysis module4004 may use the received image data to identify restocking events,low-facing events, facing events and low-stock events in at least aportion of a retail shelving unit dedicated to products from a specificproduct type. Pattern determination module 4006 may determine the demandpattern for products from the specific product type based on theidentified events. Low-stock prediction module 4008 may predict, basedon additional image data and the determined demand pattern, when thenext low-stock event will occur. Alert generation module 4010 maydetermine when an how to alert a store employee about a predicted nextlow-stock event.

In some embodiments, memory device 4000 may be a part of system 100 suchas, for example, memory device 226. Alternatively, memory device 4000may be stored in an external database or an external storagecommunicatively coupled with server 135, such as one or more databasesor memories accessible over communication network 150. Further, in otherembodiments, the components of memory device 4000 may be distributed inmore than one server and more than one memory device.

In some embodiments, retail store communication module 4002 may receiveinformation from sensors located in retail store 105. In one example,retail store communication module 4002 may receive images captured by aplurality of image sensors fixedly mounted in retail store 105 orreceive data derived from images captured by a plurality of imagesensors fixedly mounted in retail store 105. In another example, retailstore communication module 4002 may receive image data (e.g., images ordata derived from images) from robotic capturing devices configured tonavigate autonomously within retail store 105 and to capture images ofmultiple types of perishable products. In yet another example, retailstore communication module 4002 may receive data from one or more shelfsensors configured to measure properties of the products placed on astore shelf. The one or more shelf sensors may include pressuresensitive pads, light detectors, weight sensors, light sensors, odorsensors, resistive sensors, ultrasonic sensors, and more.

In some embodiments, captured data analysis module 4004 may process theinformation collected by retail store communication module 4002 toidentify restocking events, low-facing events, facing events, andlow-stock events associated with retail shelving units in retail store105. Consistent with the present disclosure, each event may beassociated with at least one criteria that captured data analysis module4004 is configured to identify. As described above, restocking eventsmay be identified when the captured data depicts a store employee hasreplenished a retail shelving unit with additional products of aspecific product type. In some cases, the image data may further depictmeans to deliver the additional products to the retail shelving unit(e.g., a cart, a pallet pump, a box, etc.). The restocking events may betriggered by low-stock events or may be initiated according to aschedule of tasks of the store employees. Low-facing events may beidentified when the captured data depicts that the number of productsplaced in a front row of products in at least a portion of the retailshelving unit is less than a facing threshold. In some cases, the facingthreshold may be determined based on the product type, the brand of theproduct, the time of day, and so forth. Facing events may be identifiedwhen the captured data depicts that the products placed on at least aportion of the retail shelving unit are reorganized by a store employeesuch that more products will be placed in a front row of the at least aportion of the retail shelving unit. The facing events may be triggeredby low-facing events or may be initiated according to a schedule oftasks of the store employees. Low-stock events may be identified whenthe captured data depicts that the number of products placed in at leasta portion of the retail shelving unit is less than a quantity threshold.Captured data analysis module 4004 may identify each of the events usingdata captured by at least one of the following types of sensors: imagesensors, pressure sensitive pads, light detectors, weight sensors, lightsensors, odor sensors, resistive sensors, and ultrasonic sensors.

In some embodiments, captured data analysis module 4004 may further beconfigured to preprocess the image data to identify the events (e.g.,restocking events, low-facing events, facing events, and low-stockevents). In some examples, the image data may be preprocessed bytransforming the image data using a transformation function to obtain atransformed image data, and the preprocessed image data may include thetransformed image data. For example, the transformation function maycomprise convolutions, visual filters (such as low-pass filters,high-pass filters, band-pass filters, all-pass filters, etc.), nonlinearfunctions, and so forth. In some examples, the image data may bepreprocessed by smoothing the image data, for example, by using Gaussianconvolution, using a median filter, and so forth. In some examples, theimage data may be preprocessed to obtain a different representation ofthe image data. For example, the preprocessed image data may include: arepresentation of at least part of the image data in a frequency domain;a Discrete Fourier Transform of at least part of the image data; aDiscrete Wavelet Transform of at least part of the image data; atime/frequency representation of at least part of the image data; arepresentation of at least part of the image data in a lower dimension;a lossy compression representation of at least part of the image data; alossless compression representation of at least part of the image data;a time order series of any of the above; any combination of the above;and so forth. In some examples, the image data may be preprocessed toextract edges, and the preprocessed image data may include informationbased on and/or related to the extracted edges. In some examples, theimage data may be preprocessed to extract visual features from the imagedata. Some examples of such visual features include edges, corners,blobs, ridges, Scale Invariant Feature Transform (SIFT) features,temporal features, and so forth. One of ordinary skill in the art willrecognize that the image data may be preprocessed using other kinds ofpreprocessing methods.

In some embodiments, pattern determination module 4006 may determine ademand pattern for products from the specific product type based on theidentified restocking event, the at least one low-facing event, the atleast one facing event, and the low-stock event. Pattern determinationmodule 4006 may determine the demand pattern based on data capturedduring the first time period. In one embodiment, pattern determinationmodule 4006 may determine the demand pattern based on at least one firsttime duration between the restocking event and the at least one facingevent, and at least one second time duration between the at least onefacing event and the low-stock event. With reference to FIG. 40A, atleast one first time duration may include the time duration between thefirst restocking event and the first facing event (i.e., between t1 tot2) and/or the time duration between the first restocking event and thesecond facing event (i.e., between t1 to t4). At least one second timeduration may include the time duration between the first restockingevent and the first low-stock event (i.e., between t2 to t6) and/or thetime duration between the second facing event and the first low-stockevent (i,e,, between t4 to t6). In another embodiment, patterndetermination module 4006 may determine the demand pattern based on thetime duration between facing events. For example, the time durationbetween the first facing event and the second facing event (i.e.,between t2 to t4). In another embodiment, pattern determination module4006 may determine the demand pattern based on the time duration betweenat least one low-facing event and the low-stock event. For example, thetime duration between the first low-facing event and the first low-stockevent (i.e., between t3 to t6) and/or the time duration between thesecond low-facing event and the first low-stock event (i.e., between t5to t6). In another embodiment, pattern determination module 4006 maydetermine the demand pattern based on a number of low-facing events orfacing events that precede the low-stock event. Generally, patterndetermination module 4006 may determine the demand pattern based on anyother relationships between identified events that occurred during thefirst period and/or any characteristics of the identified events thatoccurred during the first period.

In some embodiments, low-stock prediction module 4008 may predict whenthe next low-stock event will occur based on additional image dataassociated with images captured during a second period of time and thedetermined demand pattern. Low-stock prediction module 4008 may receivefrom captured data analysis module 4004 data about events that tookplace during the second time period,. The received data may includeimage data and may also include data from other shelf sensors. Forexample and with reference to FIG. 40A, low-stock prediction module 4008may receive data about the second restocking event that took place att7, the third low-facing event that took place at t8, the third facingevent that took place at t9, the fourth low-facing event that took placeat t10, the fourth facing event that took place at t11, the fifthlow-facing event that took place at t13, and the third restocking eventthat took place at t14. Consistent with the present disclosure,low-stock prediction module 4008 may determine, based on at least someof the identified events, a predicted time for the next low-stock event.In one embodiment, low-stock prediction module 4008 may predict that atime duration between the second restocking event that took place at t7and the predicted next low-stock event will be greater than the timeduration between the first restocking event that took place at t11 andthe low-stock event that took place at t6. In another embodiment,low-stock prediction module 4008 may predict that a time durationbetween the second restocking event that took place at t7 and thepredicted next low-stock event will be less than the time durationbetween the first restocking event that took place at t1 and thelow-stock event that took place at t6. Consistent with embodiments ofthe present disclosure, low-stock prediction module 4008 may predictwhen the next low-stock event will occur based on image data and/oradditional information retrieved from one or more sources. For example,the additional information may include one or more of checkout data(e.g., the number of sold items during the first period of time),calendar data (e.g., holidays, sport games, etc.), historical productturnover data (e.g., the number of turkeys sold per day during the threeweeks prior to Thanksgiving of last year), and average turnover (e.g,,per hour in a day, per day in a week, per special events), and more.

In some embodiments, alert generation module 4010 may determine whetheran alert associated with the predicted next low-stock event should begenerated and to generate alerts when needed. The term “alert” refers toany text message, voice message, notification, or other data provided bythe system to an entity associated with the retail store about apotential low-stock event in at least a portion of the retail shelvingunit that should be replenished with additional products of a specificproduct type associated with the at least a portion of the retailshelving unit. In one embodiment, alert generation module 4010 mayobtain inventory data associated with the specific product type. Basedon the inventory data, alert generation module 4010 may determine towithhold the alert associated with the predicted next low-stock event.For example, there is no reason to generate an alert associated with thepredicted next low-stock event when there are not available productsfrom the specific product type to replenish at least a portion of theretail shelving unit. When the alert generation module 4010 determinesto withhold the alert associated with the predicted next low-stockevent, the system may initiate an alternate action to change thearrangement of products on the retail shelving unit. For example, thealternate action may include instructing a store employee to placeproducts from a differing product type on the at least a portion of aretail shelving unit. In another embodiment, alert generation module4010 may obtain schedule data associated with scheduled restockingevents. Based on the schedule data, alert generation module 4010 maydetermine to forgo providing the alert associated with the predictednext low-stock event. For example, there may be little value ingenerating an alert associated with the predicted next low-stock eventwhen a store employee is scheduled to replenish the at least a portionof the retail shelving unit in the next five minutes.

In some embodiments, database access module 4012 may cooperate withdatabase 4014 to retrieve information such as product supplyinformation. The product supply information may include, for example,one or more of a schedule providing arrival dates and/or times ofadditional products, inventory records, checkout data, calendar data,and historical product turnover data. Alert generation module 4010 mayuse the product supply information stored in database 4014 whendetermining whether to issue or to withhold an alert associated with thepredicted next low-stock event. Database 4014 may include separatedatabases, including, for example, a vector database, raster database,tile database, viewport database, and/or a user input database,configured to store data. The data stored in database 4014 may bereceived from modules 4002-4012, server 135, from any communicationdevice associated with retail stores 105. Moreover, the data stored indatabase 4014 may be provided as input using data entry, data transfer,or data uploading.

Modules 4002-4012 may be implemented in software, hardware, firmware, amix of any of those, or the like. For example, if the modules areimplemented in software, the modules may be stored in a server (e.g.,server 135) or distributed over a plurality of servers. In someembodiments, any one or more of modules 4002-4012 and data associatedwith database 4014 may be stored in database 140 and/or located onserver 135, which may include one or more processing devices. Processingdevices of server 135 may be configured to execute the instructions ofmodules 4002-4012. In some embodiments, aspects of modules 4002-4012 mayinclude software, hardware, or firmware instructions (or a combinationthereof) executable by one or more processors, alone, or in variouscombinations with each other. For example, modules 4002-4012 may beconfigured to interact with each other and/or other modules of server135 to perform functions consistent with disclosed embodiments.

FIG. 41 depicts an exemplary method 4100 for generating low-stock alertsbefore a low-stock event is predicted to occur. For purposes ofillustration, in the following description, reference is made to certaincomponents of system 100. It will be appreciated, however, that otherimplementations are possible and that other components may be utilizedto implement the exemplary method. It will also be readily appreciatedthat the illustrated method can be altered to modify the order of steps,delete steps, or further include additional steps.

At step 4102, a processing device (e.g., processing device 202) mayreceive image data associated with images captured during a first periodof time, wherein the captured data depicts inventory changes in at leasta portion of a retail shelving unit dedicated to products from aspecific product type. Consistent with the present disclosure, the firsttime period may be longer than a day, longer than a week, or longer thana month. In one embodiment, the image data may be acquired by any one ofimage capturing devices 125 illustrated in FIG. 4A-4C. For example, theimage data may be acquired by a plurality of image sensors fixedlymounted in retail store 105. Details of the image capturing devicesmounted in retail store 105 and configured to acquire the imagescaptured during a first period of time are described with references toFIGS. 5A-5C. The image data may include processed images (e.g,, croppedimages, video streams, and more) or information derived from thecaptured images (e.g., data about events identified in captured images,data that may be used to construct a 3D image, and more). In oneembodiment, the processing device may periodically receive images thatcorrespond with the current inventory of the products from the specificproduct type. For example, the processing device may receive one or moreimages at predetermined time intervals (e.g., every minute, every 5minutes, every 15 minutes, every 30 minutes, etc.). In anotherembodiment, the processing device may receive the one or more images inresponse to a detection of an identified event. In one example, theprocessing device may receive the one or more images after an event,such as a detected lifting of a product from the specific product typefrom the shelving unit.

At step 4104, the processing device may analyze the image data toidentify a restocking event associated with the at least a portion ofthe retail shelving unit. In one embodiment, identifying the restockingevent may include detecting a store employee replenishing the at least aportion of the retail shelving unit with additional products of thespecific product type. For example, as illustrated in FIG. 39A, storeemployee 3902 (or other individual) may replenish store shelf 3900 withadditional products from cart 3904. In disclosed embodiments,identifying the restocking event may include detecting a store employeereplenishing at least a portion of the retail shelving unit withadditional products of the specific product type. Moreover, theprocessing device may be configured to identify a time of day that thestore employee replenished at least a portion of the retail shelvingunit. In some cases, the identified time of day may later be used fordetermining a demand pattern for products of the specific-product type.

At step 4106, the processing device may analyze the image data toidentify at least one facing event associated with the at least aportion of the retail shelving unit, wherein the at least one facingevent occurs after the restocking event. In one embodiment, identifyingthe at least one facing event may include detecting a store employeereorganizing products from the specific product type such that moreproducts out of the products already placed on the at least a portion ofthe retail shelving unit will be placed in a front row of the at least aportion of the retail shelving unit. For example, as illustrated in FIG.39C, store employee 3906 (or other individual) may replenish at least aportion of the retail shelving unit with products already on store shelf3900. In some embodiments, identifying the at least one facing event mayinclude detecting a store employee reorganizing products from thespecific product type such that more products out of the productsalready placed on the at least a portion of the retail shelving unitwill be placed in a front row or the at least a portion of the retailshelving unit. Moreover, the processing device may be configured toassign a facing level for the at least one facing event or for thelow-facing event. The facing level may represent the percentage ornumber of products placed in a front row of the at least a portion ofthe retail shelving unit. In some cases, the facing level for the atleast one facing event or for the at least one low-lacing event may beused later for determining the demand pattern.

At step 4108, the processing device may analyze the image data toidentify a low-stock event associated with the at least a portion of theretail shelving unit, wherein the low-stock event occurs after the atleast one facing event. In one embodiment, identifying the low-stockevent includes detecting that a number of products from the specificproduct type in the at least a portion of the retail shelving unit isless than threshold. For example, as illustrated in FIG. 39F, thethreshold may be a single product in the front row of store shelf 3900.In some embodiments, identifying the low-stock event includes detectingthat a number of products from the specific product type in the at leasta portion of the retail shelving unit is less than a quantity threshold.Moreover, the processing device may be configured to determine a typicaltime duration from the identification of the low-stock event of productsfrom the specific product type until the at least a portion of theretail shelving unit is replenished. In sonic cases, the determined timeduration may be used later in order to determine a time for issuing thealert.

At step 4110, the processing device may determine a demand pattern forproducts from the specific product type based on the identifiedrestocking event, the at least one facing event, and the low-stockevent. Consistent with the present disclosure, the processing device mayperform big data analytics on the image data and the identified events.Typically, the big data analytics are executed in an automatic orautonomous manner without using any user input during the learninganalysis. In one embodiment, the processing device may performlarge-scale data analysis on the received information to discover,detect, or learn new data. For example, the large-scale data analysismay involve the process of examining large quantities of data (e.g.,data mining, data discovery, etc.) to extract new or previously unknowninteresting data or patterns such as unusual demand times for thespecific product type. The processing device may also performlarge-scale data analysis on the stored data (e.g., machine learninganalysis, data modeling, pattern recognition, predictive analysis,correlation analysis, etc.) to predict, calculate, or identify implicitrelationships or inferences within the stored data. For example, thedetermined demand pattern may correlate a combination of differentparameters to product sales and/or to depletion of product from retailshelves. The different parameters may include the quantity of productson display, the predicted demand for the products, the location that theproducts are being displayed, the time of day, the day of the week, theprice of products, and more. For example, when the first time period islonger than a week, the determined pattern may take into account certaindays of the week with higher demand for the specific product type. Indisclosed embodiments, the processing device may obtain statistical dataindicative of a number of consumers in the retail store during differenthours of a day and determine the demand pattern based on the statisticaldata. In some embodiments, the processing device may take into accountthe current number and the expected number of consumers in the retailstore when determining a time for issuing the alert such that the nextrestocking event would take place before the quantity of products fromthe specific product type reaches to the quantity threshold. In someexamples, the determined demand pattern may include a stochastic model(such as a random walk model, a Markov model, Levy model, and so forth)for predicting product sales and/or depletion of product from retailshelves based on the different parameters. In some examples, thedetermined demand pattern may include a machine learning model trainedusing training examples to predict product sales and/or depletion ofproduct from retail shelves based on the different parameters. Forexample, at least one of the training examples may be based on pastinformation collected from the retail store as described above. Inanother example, at least one of the training examples may be based oninformation collected from the other retail stores.

At step 4112, the processing device may receive additional image dataassociated with images captured during a second period of time,subsequent to the low-stock event, by the plurality of image capturingdevices. The additional image data may depict additional inventorychanges in the at least a portion of the retail shelving unit.Consistent with the present disclosure, the processing device mayanalyze the additional image data associated with images captured duringthe second period to detect a second low-stock event that happenedearlier than a predicted time of the predicted low-stock event. Whenthis happens, the processing device may use a machine learning module orother data gathering algorithm to update the demand pattern with thesecond low-stock event. For example, the predicted time of the predictedlow-stock event was based on the predicted number of customers in thestore and the actual number of customers was higher. In addition, theprocessing device may estimate a number of products picked up from theat least a portion of the retail shelving unit during the second periodof time to improve a certainty level for the predicted next low-stockevent.

At step 4114, the processing device may predict when the next low-stockevent will occur based on the additional image data and the determineddemand pattern. In some embodiments, the processing device may determinea confidence level associated with the predicted next low-stock event.When the confidence level is lower than a threshold, the processingdevice may withhold the alert associated with the predicted nextlow-stock event, and initiate an action to increase the confidencelevel. In some cases, the action to improve the confidence level mayinclude instructing a store employee to count a current number ofproducts from the specific product type located on the at least aportion of the retail shelving unit. Consistent with the presentembodiments, the threshold for the confidence level associated with thepredicted next low-stock event may be depend on the type of products(e.g., products stored in cold rooms may be associated with higherthreshold for the confidence level than other products). Additionally,the threshold for the confidence level associated with the predictednext low-stock event may be depend on the time of day (e.g., in themornings, the threshold for the confidence level may be lower than inthe afternoons).

At step 4116, the processing device may generate an alert associatedwith the predicted next low-stock event. In some embodiments, theprocessing device may obtain inventory data associated with the specificproduct type. Based on the inventory data, the processing device maydetermine to withhold the alert associated with the predicted nextlow-stock event. For example, the inventory data may indicate that thereare no additional products from the specific product type in storage,thus there is no point in sending the alert. The processing device mayinitiate an alternate action to change the arrangement of products onthe retail shelving unit. For example, the alternate action may includeinstructing a store employee to place products front a differing producttype on at least a portion of a retail shelving unit. In otherembodiments, the processing device may obtain schedule data associatedwith scheduled restocking events. The schedule data may include detailson the status of restocking task. Based on the schedule data, theprocessing device may determine to forgo providing the alert associatedwith the predicted next low-stock event. In addition, the processingdevice may analyze the additional image data associated with imagescaptured during the second period to detect a second low-stock eventthat happened before the restocking event triggered by the system. Whenthis happens, the processing device may use a machine learning module orother algorithm to update the timing of the alert. For example, it maytake 10 minutes to obtain additional products of a first product typefrom a storage location and about 20 minutes to obtain additionalproducts of a second product type from a storage location,

Embodiments of the present disclosure may allow for a system to processimages captured in a retail store and automatically identify productsdisplayed on shelving units. For example, the system may receive a firstimage depicting a shelving unit with a plurality of products ofdiffering product types displayed thereon and a first plurality oflabels coupled to the shelving unit. Each label depiction in the firstimage may have at most a first image resolution. The system may receivea set of second images depicting a second plurality of labels. Eachlabel depiction in the set of second images may at least a second imageresolution greater than the first image resolution. The system maycorrelate a first label depiction of a specific label included in thefirst image to a second label depiction of the same specific labelincluded in a second image. The second image may be included in the setof second images. The system may use information derived from the secondlabel depiction to determine a type of products displayed in the firstimage in proximity to the specific label. The system may initiate anaction based on the determined type of products displayed in proximityto the specific label.

FIG. 42 is a schematic illustration of an example system 4200 forprocessing images captured in a retail store and automaticallyidentifying products displayed on shelving units, consistent with theembodiments of the present disclosure. As illustrated in FIG. 42, system4200 may include a computing device 4201, database 4202, one or moreimage capturing devices 4203A, 4203B, 4203C, . . . , 4203N (or capturingdevice 4203, collectively), and network 4205.

Computing device 4201 may be configured to receive one or more imagescaptured in a retail store. Computing device 4201 may receive the imagesfrom capturing device 4203 directly or via network 4205. Alternatively,the one or more images captured in the retail store may be prestored indatabase 4202, and computing device 4201 may retrieve the images fromdatabase 4202 directly or via network 4205.

Computing device 4201 may also be configured to process the one or moreimages captured in the retail store to automatically identify productsdisplayed on shelving units. Computing device 4201 may use any suitableimage analysis technique including, for example, object recognition,object detection, image segmentation, feature extraction, opticalcharacter recognition (OCR), object-based image analysis, shape regiontechniques, edge detection techniques, pixel-based detection, artificialneural networks, convolutional neural networks, etc. In addition,computing device 4201 may use classification algorithms to distinguishbetween different products in the retail store. In some embodiments,computing device 4201 may utilize suitably trained machine learningalgorithms and models to perform the product identification. Forexample, computing device 4201 may derive information from a labeldepicted in one of the images captured in the retail store, and use thederived information to determine a type of the products displayed inproximity to the label.

Computing device 4201 may also be configured to initiate an action basedon the determined type of the products displayed in proximity to thelabel. For example, computing device 4201 may determine whether thearrangement of the products on the shelving units comply with apredetermined planogram. If the arrangement of the products on theshelving units does not comply with a predetermined planogram, computingdevice 4201 may issue a notification to a store employee. For anotherexample, computing device 4201 may use information derived from the oneor more images to determine a displayed price associated with a specificlabel, and determine whether the type of the products displayed inproximity to the specific label is consistent with the displayed price.If the type of the products displayed in proximity to the specific labelis inconsistent with the displayed price, computing device 4201 mayissue a notification to a store employee.

Database 4202 may be configured to store information and/or data for oneor more components of system 4200. For example, database 4202 may beconfigured to store one or more images captured by capturing device4203. Computing device 4201 may access the one or more images stored indatabase 4202. As another example, database 4202 may store informationrelating to a retail store, information relating to one or more storeshelves of a retail store, information relating to one or more products,information relating to one or more non-employee individuals, or thelike, or a combination thereof. Computing device 4201 may accessinformation stored in database 4202. For example, computing device 4201may access information relating to one or more properties of a storeshelf, the physical location of the store shelves, one or more productsassociated with the store shelf.

Image capturing device 4203 may be any device configured to acquireimage data representative of products displayed in the retail store.Examples of capturing devices may include a digital camera, atime-of-flight camera, a stereo camera, an active stereo camera, a depthcamera, a Lidar system, a laser scanner, CCD based devices, or any othersensor based system capable of converting received light into electricsignals. In one embodiment, capturing device 4203 (e.g., capturingdevices 4203A and 4203N shown in FIG. 42) may include a stationarycamera with communication layers (e.g., a dedicated camera fixed to astore shelf, a security camera, etc.). The stationary cameras may befixedly mounted on in the retail store (e.g., on an opposing retailshelving unit). In another embodiment, capturing device 4203 (e.g.,capturing devices 4203B and 4203C shown in FIG. 42) may include ahandheld device (e.g,, a smartphone, a tablet, a mobile station, apersonal digital assistant, a laptop, a digital camera, and more) or awearable device (e.g., smart glasses, a smartwatch, a clip-on camera).Capturing devices 4203B and 4203C may be associated with human storeemployee 4204B and 4204C or a robot store employee (or with otherindividuals).

Network 4205 may be configured to facilitate the communication betweenthe components of system 1200. For example, computing device 4201 mayreceive the one or more images captured in the retail store fromcapturing device 4203 via network 4205. Network 4205 may include a localarea network (LAN), a wide area network (WAN), portions of the Internet,an intranet, a cellular network, a short-ranged network (e.g., aBluetooth™ based network), or the like, or a combination thereof.

In some embodiments, the images captured by capturing devices 4203A,4203B, 4203C, . . . , 4203N may have various image resolutions. The term“image resolution” is a measure of the degree to which the imagerepresents the fine details of a captured retail object (i.e., aproduct, a label, promotion, etc.). The quality of a digital image maybe determined by the total number of pixels and the range of brightnessvalues available for each pixel.

The images captured by stationary cameras (e.g., capturing devices 4203Aand 4203N) fixedly mounted on in the retail store (e.g., on an opposingretail shelving unit) may have a relatively low resolution, to theextent that the barcode and the item description on the labels may betoo small for recognition from images captured by the stationarycameras. Therefore, in some embodiments of the present disclosure,system 4200 may use two types of images. One type of images (hereinafterreferred to as “first images”) may depict a plurality of products and aplurality of labels. The first images are captured by using, forexample, from capturing devices 4203A and 4203N fixedly mounted in theretail store. The first images may have a first image resolution.Another type of images (hereinafter referred to as “second images”) maydepict labels. The second images may be captured by using, for example,capturing devices associates with a human store employee, a robot storeemployee, or a dedicated stationary image sensor. The second images mayhave a second and enhanced image resolution such that the barcode andtext (e.g., product ID, item descriptions) on the labels depicted in thesecond images may be recognized by computing device 4203. In oneembodiment, the capturing devices for capturing the second images mayinclude m*n pixels. For example, a capturing device for capturing thesecond images may have an 8 MP image sensor that includes an array of3280*2464 pixels. Each pixel may include at least one photo-voltaic cellthat converts the photons of the incident light to an electric signal.The electrical signal may be converted to digital data by an A/Dconverter and processed by an image processor (1SP). In one embodiment,the image sensor of the capturing device for capturing the second imagesmay be associated with a pixel size of between 1.1×1.1 um² and 1.7×1.7um², for example, 1.4×1.4 um².

FIG. 43A is a schematic illustration of an exemplary first image 4300,consistent with the embodiments of the present disclosure. As shown inFIG. 43A, first image 4300 may depict a shelfing unit 4310, a pluralityof products 4320 a, 4320 b, 4320 c, 4320 d, 4320 e of different types,that are displayed on shelfing unit 4310, and a plurality of labels 4330a, 4330 b, 4330 c, 4330 d, 4330 e, that are coupled to shelling, unit4310. Each type of the plurality of products 4320 a, 4320 b, 4320 c,4320 d, 4320 e may be associated with one of the plurality of labels4330 a, 4330 b, 4330 c, 4330 d, 4330 e. For example, products 4320 a areassociated with label 4330 a; products 4320 b are associated with label4330 b; and so on.

First image 4300 may have a first image resolution. In some embodiments,the first image resolution may be insufficient to identify the type ofproducts displayed in first image 4300 above a predetermined threshold.In some examples, information (e.g., bar code, product price, productID) on labels 4330 a, 4330 b, 4330 c, 4330 d, 4330 e depicted in firstimage 4300 having the first image resolution may not be readable in thefirst image resolution.

FIG. 43B is a schematic illustration of an exemplary second image 4350,consistent with the embodiments of the present disclosure. As shown inFIG. 43B, second image 4350 may depict a label 4360. Label 4360 mayinclude information regarding a type of product depicted in first image4300. The information may include, but not limited, a product ID 4362, aproduct price 4364, a visual code 4366, etc. Visual code 4366 may be avisual code (such as Barcode or QR-code) associated with product ID4362. When scanned by a code scanner, the visual code (such as theBarcode or the QR-code) may provide information related to a type of aproduct. Label 4360 depicted in second image 4350 may be associated withan area in first image 4300. For example, label 4360 in second image4350 may be associated with label 4330 in area 4340 in first image 4300.

Second image 4350 may have a second image resolution. In someembodiments, label 4360 in second image 4350 may have at least 2 times,5 times, 10 times, or 30 times more pixels than label 4330 c in firstimage 4300. For example, the first image resolution of the capturedimages may be referred to hereinafter as low-resolution and may have avalue or range of values lower than 5 megapixels, lower than, 2megapixels, lower than 1 megapixel. For example, the second imageresolution may be referred to hereinafter as high-resolution and mayhave a value or range of values higher than 2 megapixels, higher than 5megapixels, higher than 10 megapixels, for example, between 10megapixels and 20 megapixels, or about 15 megapixels. In someembodiments, the second image resolution associated with the secondlabel depiction in second image 4350 may be at least ten times the firstimage resolution associated with the first label depiction in firstimage 4300.

FIG. 44 includes a flowchart representing an exemplary method 4400 forprocessing images captured in a retail store, in accordance with exampleembodiments of the present disclosure. Method 4400 may be performed by aprocessor at a server (e.g., server 135) or a computer (e.g., one ofdevices 145A, 145B, 145C, and 145D, or computing device 4201). Forpurposes of illustration, in the following description, reference ismade to certain components of images shown in FIGS. 42, 43A, and 43B. Itwill be appreciated, however, that other implementations are possibleand that other configurations may be utilized to implement the exemplarymethod. It will also be readily appreciated that the illustrated methodcan be altered to modify the order of steps, delete steps, or furtherinclude additional steps.

At step 4402, method 4400 may include receiving a first image 4300depicting a shelving unit 4310 with products 4320 a, 4320 b, 4320 c,43:20 d, 4320 e of differing product types displayed thereon and a firstplurality of labels 4330 a, 4330 b, 4330 c, 4330 d, 4330 e coupled toshelving unit 4310. Each label depiction in first image 4300 may have atmost a first image resolution. First image 4300 may be captured by astationary camera (e.g., capturing devices 4203A or 4203N) fixedlymounted on in the retail store (e.g., on an opposing retail shelvingunit). Alternatively, first image 4300 may be captured by a capturingdevice associated with store employee (e.g., a human store employee, ora robot store employee). First image 4300 may be read from memory (e.g.,database 4202), may be received through a communication network (e,g.,network 4205) using a communication device (e.g., network interface 206in FIG. 2), may be received from an external device e.g., one of devices145A, 145B, 145C, and 145D in FIG. 1).

At step 4404, method 4400 may include receiving a set of second images4350 depicting a second plurality of labels 4360. Each label depictionin the set of second images 4350 may have at least a second imageresolution greater than the first image resolution. The second pluralityof images 4350 depicting the plurality of labels 4360 may be captured bya capturing device associated with store employee (e.g., a capturingdevice carried by a human store employee or a robot store employee.Alternatively, the second plurality of images 4350 may be captured by adedicated stationary image sensor that is configured to produce an imagehaving a resolution greater than the first image resolution. The secondplurality of images 4350 may be read from memory (e.g., database 4202),may be received through a communication network (e.g., network 4205)using a communication device (e.g., network interface 206 in FIG. 2),may be received from an external device (e.g., one of devices 145A,145B, 1450, and 145D in FIG. 1).

At step 4406, method 4400 may include correlating a first labeldepiction of a specific label included in first image to a second labeldepiction of the same specific label included in a second image includedin the set of second images. For example, the processor may correlatelabel 4330 c depicted in first image 4300 illustrated in FIG. 43A tolabel 4360 depicted in second image 4350 illustrated in FIG. 43B. Insome examples, step 4406 may include estimating a low resolutiondepiction of label 4360 (for example, by blurring the depiction of label4360 in second image 4350, by sub-sampling pixels from the depiction oflabel 4360 in second image 4350, and so forth), and comparing theestimated low resolution depiction with the depiction of label 4330 e inthe first image. For example, the estimated low resolution depiction andthe depiction of label 4330 c may be compared using an image similarityfunction to obtain a similarity score, the similarity score may becompared with a selected threshold, and step 4406 may determine acorrelation between label 4330 c and label 4360 based on a result of thecomparison (e.g., determine that the two correlates when the similarityscore is below the selected threshold). In some examples, a machinelearning model may be trained using training examples to determinewhether two depictions of labels are depictions of the same real worldlabel, and step 4406 may use the trained machine learning model toanalyze the depiction of label 4330 c in first image 4300 and thedepiction of label 4360 in second image 4350 to determine if the twodepiction correlates to the same real world label. An example of suchtraining example may include a high resolution image of a label and alow resolution image of a label, together with a label indicatingwhether the two images depicts the same real world label.

In some embodiments, step 4406 may include identifying at least onefeature in first image 4300 other than the first label depiction, andidentifying the at least one feature in the set of second images forcorrelating the first label depiction to the second label depiction. Theat least feature may include, but not limited to, color, pattern, shape,size, etc., of shelfing unit 4310 or products 4320 a, 4320 b, 4320 c,4320 d, and 4320 e, or the relative positions between the shelfing unit4310, products 4320 a, 4320 b, 4320 c, 4320 d, 4320 e, and labels 4330a, 4330 b, 4330 c, 4330 d, 4330 e. For example, the processor mayanalyze first image 4300 by using any suitable image analysis techniqueincluding, for example, object recognition, object detection, imagesegmentation, feature extraction, optical character recognition (OCR),object-based image analysis, shape region techniques, edge detectiontechniques, pixel-based detection, artificial neural networks,convolutional neural networks, etc. As a result of the image analysis onfirst image 4300, the processor may identify a color of a product inproximity to a specific label 4330 c depicted in first image 4300.Computing device 4201 may then search, among the set of second images4350, for an image that depicts at least a portion of a product that hasthe same color as the color identified in first image 4300. Once suchimage is found, computing device 4201 may correlate the depiction of thespecific label 4330 c in first image 4300 with the label (e.g., label4360) in second image 4350.

In some embodiments, step 4406 may include correlating the labeldepictions in the first and second images based on the positions of thelabel depictions. For example, a positioning sensor may also beintegrated with, or connected to, capturing devices 4203A, 4203B, 4203C,. . . , 4203N. Such positioning sensor may be implemented using one ofthe following technologies: Global Positioning System (UPS), GLObalNAvigation Satellite System (GLONASS), Galileo global navigation system,BeiDou navigation system, other Global Navigation Satellite Systems(GNSS), Indian Regional Navigation Satellite System (IRNSS), LocalPositioning Systems (LPS), Real-Time Location Systems (RTLS), IndoorPositioning System (IPS), Wi-Fi based positioning systems, cellulartriangulation, and so forth. The positioning sensor may be built into amobile capturing device, such as smartphone devices 4203B and 4203C.Alternatively, position software may allow mobile capturing devices touse internal or external positioning sensors (e.g., connecting via aserial port or Bluetooth). The processor may determine a capturingposition and orientation of first image 4300 based on positioninginformation obtained from the positioning sensor that is integrated intothe capturing device which captures first image 4300. The processor maythen determine a real world position of a specific label (e.g,, label4330 c) depicted in first image 4300 based on a relative position of thespecific label within first image 4300, and the capturing position andorientation of first image 4300 determined based on the positioninginformation obtained from the positioning sensor. Similarly, theprocessor may determine a real world position of label 4360 depicted insecond image 4350. Further, in some examples, the processor maycorrelate a specific label (e.g., label 4330 c) depicted in first image4300 to label 4360 depicted in second image 4350 based on a correlationof the determined real world position of the specific label anddetermined real world position of label 4360. For example, the processormay correlate the two labels when the difference between the determinedreal world positions of the two labels is below a selected threshold.The selected threshold may be selected manually, may be determined basedon estimated accuracy of the real world position determination process,and so forth. In another example, the process may correlate label 4360to the label of the first image that has the nearest determined realworld position to the determined real world position of label 4360.

At step 4408, method 4400 may include deriving information from thesecond label depiction, and using information derived from the secondlabel depiction to determine a type of products displayed in first image4300 in proximity to the specific label 4330 c. In this step, it isassumed that the products that are arranged in proximity to the specificlabel 4330 c are related to the specific label 4330 c.

In some embodiments, step 4408 may include analyzing second image 4350to derive information from the second label depiction. For example, theprocessor may analyze second image 4350 by using any suitable imageanalysis technique. As discussed above the suitable image analysistechnique may include, for example, object recognition, objectdetection, image segmentation, feature extraction, optical characterrecognition (OCR), object-based image analysis, shape region techniques,edge detection techniques, pixel-based detection, artificial neuralnetworks, convolutional neural networks, etc. As a result of the imageanalysis on second image 4350, the processor may recognize some textassociated with a particular meaning on label 4360 second image 4350.Based on the recognized text, the processor may determine the productID, product price, and product description (e.g., size, volume, color,brand name, manufacturer) shown on label 4360. Based on the product IDand product description, the processor may determine the type of theproducts that are arranged in proximity to the specific label 4330 c.

In some embodiments, step 4408 may include performing an opticalcharacter recognition (OCR) process on the second label depiction todetermine the type of products in first image 4300 displayed inproximity to the specific label 4330 c. For example, the processor mayperform an OCR process on second label 4360 to recognize some text. Theprocessor may compare the recognized text against the text in a productinventory or catalog associated with the retail store and stored in adatabase (e.g., database 4202), and may identify a product ID or productdescription that matches the recognized text.

In some embodiments, step 4408 may include processing a visual codeshown in the second label depiction to determine the type of productsdisplayed in first image 4300 in proximity to the specific label 4330 c.For example, the processor may identify a visual code 4366 (such as aBarcode or a QR-code) shown in second label 4360, and retrieveinformation associated with the visual code (such as the Barcode or theQR-code) from a local database or From au external database (e.g.,database 4202) via network 4205. The information associated with thevisual code (such as the Barcode or the QR-code) may include a producttype, a product ID, a product price, and/or a product description (e.g.,size, volume, color, brand name, manufacturer, and so forth).

In some examples, step 4408 may include determining the type of productdisplayed in an area in a first image based, at least in part, on datacollected from multiple retail stores. For example, the processor mayanalyze an image captured in another retail store that is different fromthe retail store being monitored. The processor may identify, in theimage of the other retail store, a label that is similar as label 4360depicted in second image 4350. The processor may then identify, in theimage of the other retail store, a type of products arranged inproximity to the similar label. The processor may determine that theidentified product type is the same as the products in area 4340 inproximity to label 4330 c, which is related to label 4360 depicted insecond image 4350.

In some examples, the type of product displayed in an area in a firstimage (e.g., area 4340 in first image 4300) may be determined based, atleast in part, on a representation of a plurality of products determinedfrom a label or a label embedded in a plastic cover that may be used fordisplaying prices.

At step 4410, method 4400 may include initiating an action based on thedetermined type of products displayed in proximity to the specificlabel.

In some embodiments, step 4410 may include determine planogramcompliance based on the type of products and data derived from at leastone first image captured at the first image resolution. A planogram maybe associated with contractual obligations and/or other preferencesrelated to the retailer methodology for placement of a type of productson the store shelves. For example, the planogram may describe a desiredplacement of a plurality of types of products on the store shelves. Thedata derived from at least one first image captured at the first imageresolution may include an actual placement of a plurality of productsrelative to a plurality of labels. The processor may retrieve at leastone planogram regarding one or more types of products from a storedatabase or an external database (database 4202) via network 4205, basedon the type of products determined based on the label depicted in thesecond image 4350. The processor may also analyze first image 4300 todetermine an actual placement of the plurality of products relative to aplurality of labels. The processor may then detect differences betweenthe at least one planogram and the actual placement of the plurality ofproducts relative to plurality of labels. The processor may determine aplanogram incompliance event based, at least in part, on detecteddifferences. Step 4410 may also include, in response to determining theplanogram incompliance event, generating and issuing a user-notificationabout the planogram incompliance event. The user-notification mayinclude information about a correct display location of a misplacedproduct, information about a store shelf associated with the misplacedproduct, information about a type of the misplaced product, and/or avisual depiction of the misplaced product.

In some embodiments, the planogram stored in the store datable or theexternal database may include a detail (e.g., product name, brand name,price, sale, etc.) of a type of products. Thus, step 4410 may includeusing information derived from the second label depiction to determine adisplayed detail associated with the type of products displayed in firstimage 4300 in proximity to the specific label 4330 c. For example, theprocessor may determine a product price displayed on a label next to theproducts based on data derived from second image 4350, and compare thelabel price with the price described by the planogram retrieved from thestore database. The processor may determine a planogram incomplianceevent based, at least in part, on one or more inconsistencies betweenthe detail displayed on the label and the detail included in theplanogram. Step 4410 may also include initiating the action based on theplanogram incompliance event. For example, the processor may correct theproduct detail in the store database or correct the label. The processormay also determine that the correlation of the first label depiction ofa specific label included in the first image to the second labeldepiction of the same specific label included in a second image wasincorrect, and/or reduce a certainty level associated with thecorrelation.

In some embodiments, the action initiated at step 4410 may includeupdating a machine learning model for product recognition. For example,when the processor determines that the correlation of the first labeldepiction of a specific label included in the first image to the secondlabel depiction of the same specific label included in a second imagewas incorrect, the processor may update a machine learning model forrecognizing products in first image 4300 based on the informationderived from the label in second image 4350. Alternatively, theprocessor may update a machine learning model for recognizing text inthe label in second image 4350 based on the product identified in firstimage 4300.

In some embodiments, step 4410 may include generating a representationof the shelving unit that includes at least one continuous area from theat least one first image and a plurality of discontinuous areas from theset of second images. For example, the continuous area may be the entirearea shown in first image 4300 in FIG. 43A, and the plurality ofdiscontinuous areas may include a plurality of labels shown in differentsecond images, including second image 4350 in FIG. 43B.

In some embodiments, step 4410 may further include using informationderived from first image 4300 to identify a group of products associatedwith a type of products identified in the specific label 4330 c, andinitiating an action when a location of the group of products relativeto the specific label 4330 c deviates from store policies. For example,when the specific label 4330 c should be below the products associatedwith the specific label 4330 c, but it is determined based on theinformation derived from first image 4300 that the specific label 4330 cis above the products, the processor may generate and issue aninstruction fora store employee to move the specific label 4330 c or theproducts.

In some embodiments, method 4400 may further include receiving aplurality of first images 4300 captured by at least one image sensorfixedly mounted on a shelving unit opposing shelving unit 4310associated with the specific label 4330 c, analyzing the plurality offirst images 4300 to identify a time when a new label is presented onshelving unit 4310, and generating an instruction for a human storeemployee to capture a second image 4350 of the new label. Additionallyor alternatively, method 4400 may include issuing a crowd sourcing taskfor a store employee or a dedicated sensor to capture a second image4350 of the new label. The instruction for the store employee to capturethe second image 4350 of the new label includes information may includea location of the new label. Alternatively, the instruction for thestore employee to capture the second image of the new label may includeat least a portion of the first image that includes the new label. Thestore employee instructed to capture the second image of the new labelmay be a robot employee or a human employee.

The foregoing description has been presented for purposes ofillustration. It is not exhaustive and is not limited to the preciseforms or embodiments disclosed. Modifications and adaptations will beapparent to those skilled in the art from consideration of thespecification and practice of the disclosed embodiments. Additionally,although aspects of the disclosed embodiments are described as beingstored in memory, one skilled in the art will appreciate that theseaspects can also be stored on other types of computer readable media,such as secondary storage devices, for example, hard disks or CD ROM, orother forms of RAM or ROM, TAB media, DVD, Blu-ray, 4K Ultra HD Blu-ray,or other optical drive media.

Computer programs based on the written description and disclosed methodsare within the skill of an experienced developer. The various programsor program modules can be created using any of the techniques known toone skilled in the art or can be designed in connection with existingsoftware. For example, program sections or program modules can bedesigned in or by means of .Net Framework, .Net Compact Framework (andrelated languages, such as Visual Basic, C, etc.), Java, C++,Objective-C, HTML, FITMUAJAX combinations, XML, or HTML with includedJava applets.

Moreover, while illustrative embodiments have been described herein, thescope of any and all embodiments having equivalent elements,modifications, omissions, combinations (e.g., of aspects across variousembodiments), adaptations and/or alterations as would be appreciated bythose skilled in the art based on the present disclosure. Thelimitations in the claims are to be interpreted broadly based on thelanguage employed in the claims and not limited to examples described inthe present specification or during the prosecution of the application.The examples are to be construed as non-exclusive. Furthermore, thesteps of the disclosed methods may be modified in any manner, includingby reordering steps and/or inserting or deleting steps. It is intended,therefore, that the specification and examples be considered asillustrative only, with a true scope and spirit being indicated by thefollowing claims and their full scope of equivalents.

1-161. (canceled)
 162. A system for processing images captured in aretail store and automatically determining options for store executionbased on shelf capacity, the system comprising: at least one processorconfigured to: receive a set of images captured during a period of time,wherein the set of images depict a first plurality of products from aparticular product type displayed on a shelving unit in a retail store;analyze the set of images to identify a portion of the shelving unitassociated with the particular product type; determine a productcapacity for the portion of the shelving unit dedicated to theparticular product type, wherein the determined product capacity isgreater than a number of the first plurality of products depicted in theset of images; access stored data about a second plurality of productsfrom the particular product type, wherein each of the second pluralityof products is located separately from the shelving unit when the periodof time ends; use the product capacity for the portion of the shelvingunit dedicated to the particular product type and the accessed data togenerate at least one suggestion for improving store execution; andprovide the at least one suggestion for improving store execution. 163.The system of claim 162, wherein the at least one processor is furtherconfigured to: obtain data indicative of a depth of the shelving unit;and determine the product capacity for the portion of the shelving unitdedicated to the particular product type based on the depth of theshelving unit.
 164. The system of claim 163, wherein the at least oneprocessor is further configured to: analyze the set of images todetermine at least one dimension of products from the particular producttype; and determine the product capacity for the portion of the shelvingunit dedicated to the particular product type based on the depth of theshelving unit and the at least one dimension of products from theparticular product type.
 165. The system of claim 64, wherein the atleast one processor is further configured to: based on the at least onedimension of products from the particular product type, determine amaximum number of products from the particular product type that can beplaced in a front row of products associated with the portion of theshelving unit dedicated to the particular product type; based on theobtained shelf depth, determine a maximum number of products from theparticular product type that can be placed behind the front row ofproducts; and determine the product capacity associated with the portionof the shelving unit dedicated to the particular product type based onthe maximum number of products that can be placed in the front row andthe maximum number of products that can be placed behind the front row.166. A computer program product for processing images captured in aretail store and automatically determining options for store executionbased on shelf capacity, the computer program product embodied in anon-transitory computer-readable medium and including instructions forcausing at least one processor to execute a method comprising: receivinga set of images captured during a period of time, wherein the set ofimages depicts a first plurality of products from a particular producttype displayed on a shelving unit in a retail store; analyzing the setof images to identify a portion of the shelving unit dedicated to theparticular product type; determining a product capacity for the portionof the shelving unit dedicated to the particular product type, whereinthe determined product capacity is greater than a number of the firstplurality of products depicted in the set of images; accessing storeddata about a second plurality of products from the particular producttype, wherein each of the second plurality of products is locatedseparately from the shelving unit when the period of time ends; usingthe determined product capacity and the accessed data to generate atleast one suggestion for improving store execution; and providing the atleast one suggestion for improving store execution.
 167. The computerprogram product of claim 166, wherein the at least one suggestion forimproving the store execution includes changing a size of the portion ofthe at least one store shelf dedicated to products from the particularproduct type.
 168. The computer program product of claim 166, whereinthe at least one suggestion for improving the store execution includeschanging a re-stocking frequency of products from the particular producttype at the at least one store shelf.
 169. The computer program productof claim 166, wherein the at least one suggestion for improving thestore execution includes at least one of: changing product assortmentfor the retail store, changing a product assortment for the shelvingunit, changing a position of products in the shelving unit, changing aspace dedicated to products from the particular product type, changing atype of shelf of the shelving unit, or changing a spacing betweenshelves of the shelving unit.
 170. The computer program product of claim166, wherein the at least one suggestion for improving the storeexecution includes at least one of: changing a storage capacity ofproducts from the particular product type in the store's backroom,changing a location of products from the particular product type in thestore's backroom, or providing an order recommendation for restockingthe store's backroom.
 171. The computer program product of claim 166,wherein the stored data includes sales data indicative of a number ofthe second plurality of products sold during the period of time, and themethod further includes: analyzing the set of images to estimate anumber of products from the particular product type placed on theportion of the shelving unit; determining a level of product populationat the portion of the shelving unit dedicated to the particular producttype based on the estimated number of products and the determinedproduct capacity; identifying changes in the level of product populationduring the period of time; and determining the at least one suggestionfor improving the store execution based on identified correlationsbetween the sales data and the changes in the level of productpopulation.
 172. The computer program product of claim 171, wherein themethod further includes: obtaining additional sales data indicative ofproducts from the particular product type sold during a second period oftime after the at least one suggestion was implemented; and determiningwhether implementing the at least one suggestion resulted in highersales of the particular product type during the second period of time.173. The computer program product of claim 166, wherein the stored dataincludes inventory data indicative of a number of the second pluralityof products available in at least one storage area associated with theretail store, and the method further includes: identifying changes in alevel of product population at the portion of the shelving unitdedicated to the particular product type during the period of time basedon the set of images and the determined product capacity; and generatingthe at least one suggestion for improving the store execution based onthe changes in the level of product population and the inventory data;and displaying the at east one suggestion to an employee of the retailstore.
 174. The computer program product of claim 166, wherein themethod further includes: comparing the determined product capacity forthe portion of the shelving unit dedicated to the particular producttype with a capacity for product facings for the particular product typeat the portion of the shelving unit dedicated to the particular producttype; in response to a first result of the comparison, providing asuggestion for changing the capacity for product facings for theparticular product type at the portion of the shelving unit dedicated tothe particular product type; and in response to a second result of thecomparison, forgoing providing the suggestion
 175. The computer programproduct of claim 166, wherein the stored data includes reference dataindicative of a number of the second plurality of products available ona shelving unit of at least one other retail store, and the methodfurther includes: identifying changes in a level of product populationat the portion of the shelving unit dedicated to the particular producttype during the period of time based on the set of images and thedetermined product capacity; and generating and providing the at leastone suggestion for improving the store execution based on the changes ina level of product population and the reference data.
 176. The computerprogram product of claim 166, wherein the method further includes:analyzing the set of images to detect changes during the period of timein a number of front facing products placed on the portion of theshelving unit dedicated to the particular product type; and using theproduct capacity and the detected changes in the number of front facingproducts to generate and display the at least one suggestion forimproving store execution.
 177. The computer program product of claim176, wherein the at least one suggestion includes changing a restockingschedule of products from the particular product type to reduce a timeduration where the number of front facing products is less than athreshold.
 178. The computer program product of claim 177, wherein themethod further includes: obtaining sales data reflective of productsfrom the particular product type sold during a second period of timeafter the at least one suggestion was implemented; and determiningwhether changing the restocking schedule to reduce the time durationwhere the number of front facing products is less than the thresholdresulted in higher sales of the particular product type during thesecond period of time.
 179. The computer program product of claim 176,wherein the method further includes: using the determined productcapacity and the detected changes in the number of front facing productsto predict a time when the number of front facing products is less thana threshold: and determining at least one suggestion for restocking theportion of the shelving unit dedicated to the particular product typebefore the predicted time.
 180. The computer program product of claim179, wherein the method further includes: causing restocking, of theportion of the shelving unit dedicated to the particular product typewhile the number of front facing products is at a maximum possiblenumber of front facing products associated with the portion of theshelving unit dedicated to the particular product type.
 181. A methodfor processing images captured in a retail store and automaticallydetermining options for store execution based on shelf capacity, themethod comprising: receiving a set of images captured during a period oftime, wherein the set of images depicts a first plurality of productsfrom a particular product type displayed on a shelving unit in a retailstore: analyzing the set of images to identify a portion of the shelvingunit dedicated to the particular product type; determining a productcapacity for the portion of the shelving unit dedicated to theparticular product type, wherein the determined product capacity isgreater than a number of the first plurality of products depicted in theset of images: accessing stored data about a second plurality ofproducts from the particular product type, wherein each of the secondplurality of products is located separately from the shelving unit whenthe period of time ends; using the determined product capacity and theaccessed data to generate at least one suggestion for improving storeexecution; and providing the at least one suggestion for improving storeexecution. 182-221. (canceled)